2022-12-01T10:12:06.1211224Z Requested labels: linux.8xlarge.nvidia.gpu
2022-12-01T10:12:06.1211341Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/89997/merge
2022-12-01T10:12:06.1211389Z Waiting for a runner to pick up this job...
2022-12-01T10:12:06.8315387Z Job is about to start running on the runner: i-0513dad604108ca34 (organization)
2022-12-01T10:12:10.4262415Z Current runner version: '2.299.1'
2022-12-01T10:12:10.4270435Z Runner name: 'i-0513dad604108ca34'
2022-12-01T10:12:10.4271142Z Runner group name: 'Default'
2022-12-01T10:12:10.4271876Z Machine name: 'ip-10-0-2-60'
2022-12-01T10:12:10.4274670Z ##[group]GITHUB_TOKEN Permissions
2022-12-01T10:12:10.4275567Z Actions: read
2022-12-01T10:12:10.4276001Z Checks: read
2022-12-01T10:12:10.4276378Z Contents: read
2022-12-01T10:12:10.4276888Z Deployments: read
2022-12-01T10:12:10.4277321Z Discussions: read
2022-12-01T10:12:10.4277689Z Issues: read
2022-12-01T10:12:10.4278110Z Metadata: read
2022-12-01T10:12:10.4278561Z Packages: read
2022-12-01T10:12:10.4278931Z Pages: read
2022-12-01T10:12:10.4279406Z PullRequests: read
2022-12-01T10:12:10.4279892Z RepositoryProjects: read
2022-12-01T10:12:10.4280312Z SecurityEvents: read
2022-12-01T10:12:10.4280778Z Statuses: read
2022-12-01T10:12:10.4281226Z ##[endgroup]
2022-12-01T10:12:10.4285321Z Secret source: None
2022-12-01T10:12:10.4286202Z Prepare workflow directory
2022-12-01T10:12:10.5595007Z Prepare all required actions
2022-12-01T10:12:10.5821573Z Getting action download info
2022-12-01T10:12:10.8160013Z Download action repository 'pytorch/pytorch@master' (SHA:850b53bbee82fb194af85b566aedee94b96def32)
2022-12-01T10:12:14.7694699Z Download action repository 'pytorch/test-infra@main' (SHA:1f415583bdcd967e33ea8fd05be71ed0bdf19880)
2022-12-01T10:12:15.0002842Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-12-01T10:12:15.1257159Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-12-01T10:12:15.4104132Z Getting action download info
2022-12-01T10:12:15.5605819Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-12-01T10:12:15.7641297Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-12-01T10:12:15.7643837Z ##[group] Inputs
2022-12-01T10:12:15.7644264Z   build-environment: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:12:15.7645450Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
]}

2022-12-01T10:12:15.7646701Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:12:15.7647184Z   sync-tag: 
2022-12-01T10:12:15.7647437Z ##[endgroup]
2022-12-01T10:12:15.7648312Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu)
2022-12-01T10:12:15.8767299Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-12-01T10:12:15.8767704Z with:
2022-12-01T10:12:15.8767954Z   submodules: recursive
2022-12-01T10:12:15.8768215Z   fetch-depth: 0
2022-12-01T10:12:15.8768430Z env:
2022-12-01T10:12:15.8768672Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:12:15.8768936Z ##[endgroup]
2022-12-01T10:12:15.9085611Z ##[group]Run retry () {
2022-12-01T10:12:15.9085947Z retry () {
2022-12-01T10:12:15.9086244Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-12-01T10:12:15.9086543Z }
2022-12-01T10:12:15.9086802Z echo "${GITHUB_WORKSPACE}"
2022-12-01T10:12:15.9087081Z if [ -z "${NO_SUDO}" ]; then
2022-12-01T10:12:15.9087396Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-12-01T10:12:15.9087888Z else
2022-12-01T10:12:15.9088185Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-12-01T10:12:15.9088468Z fi
2022-12-01T10:12:15.9088772Z mkdir "${GITHUB_WORKSPACE}"
2022-12-01T10:12:15.9107120Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:12:15.9107437Z env:
2022-12-01T10:12:15.9107689Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:12:15.9107954Z   NO_SUDO: 
2022-12-01T10:12:15.9108177Z ##[endgroup]
2022-12-01T10:12:15.9347063Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:12:15.9778537Z ##[group]Run malfet/checkout@silent-checkout
2022-12-01T10:12:15.9778975Z with:
2022-12-01T10:12:15.9779257Z   ref: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:12:15.9779545Z   fetch-depth: 0
2022-12-01T10:12:15.9779781Z   submodules: recursive
2022-12-01T10:12:15.9780039Z   quiet-checkout: true
2022-12-01T10:12:15.9780310Z   repository: pytorch/pytorch
2022-12-01T10:12:15.9780760Z   token: ***
2022-12-01T10:12:15.9781002Z   ssh-strict: true
2022-12-01T10:12:15.9781268Z   persist-credentials: true
2022-12-01T10:12:15.9781528Z   clean: true
2022-12-01T10:12:15.9781745Z   lfs: false
2022-12-01T10:12:15.9781994Z   set-safe-directory: true
2022-12-01T10:12:15.9782239Z env:
2022-12-01T10:12:15.9782456Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:12:15.9782711Z ##[endgroup]
2022-12-01T10:12:16.1361221Z Syncing repository: pytorch/pytorch
2022-12-01T10:12:16.1363081Z ##[group]Getting Git version info
2022-12-01T10:12:16.1363632Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-12-01T10:12:16.1364210Z [command]/usr/bin/git version
2022-12-01T10:12:16.1364494Z git version 2.37.1
2022-12-01T10:12:16.1373359Z ##[endgroup]
2022-12-01T10:12:16.1392970Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/65f6091c-7401-4092-a3a7-81fee12c4235' before making global git config changes
2022-12-01T10:12:16.1394190Z Adding repository directory to the temporary git global config as a safe directory
2022-12-01T10:12:16.1400139Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:12:16.1445227Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-12-01T10:12:16.1451576Z ##[group]Initializing the repository
2022-12-01T10:12:16.1455449Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:12:16.1487309Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-12-01T10:12:16.1487778Z hint: is subject to change. To configure the initial branch name to use in all
2022-12-01T10:12:16.1488216Z hint: of your new repositories, which will suppress this warning, call:
2022-12-01T10:12:16.1488545Z hint: 
2022-12-01T10:12:16.1488872Z hint: 	git config --global init.defaultBranch <name>
2022-12-01T10:12:16.1489163Z hint: 
2022-12-01T10:12:16.1489543Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-12-01T10:12:16.1490031Z hint: 'development'. The just-created branch can be renamed via this command:
2022-12-01T10:12:16.1490362Z hint: 
2022-12-01T10:12:16.1490800Z hint: 	git branch -m <name>
2022-12-01T10:12:16.1491325Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-12-01T10:12:16.1501880Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-12-01T10:12:16.1536204Z ##[endgroup]
2022-12-01T10:12:16.1536697Z ##[group]Disabling automatic garbage collection
2022-12-01T10:12:16.1540508Z [command]/usr/bin/git config --local gc.auto 0
2022-12-01T10:12:16.1571003Z ##[endgroup]
2022-12-01T10:12:16.1572010Z ##[group]Setting up auth
2022-12-01T10:12:16.1581404Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-12-01T10:12:16.1614678Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-12-01T10:12:16.1941021Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-12-01T10:12:16.1971457Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-12-01T10:12:16.2270584Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-12-01T10:12:16.2315179Z ##[endgroup]
2022-12-01T10:12:16.2315659Z ##[group]Fetching the repository
2022-12-01T10:12:16.2323924Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-12-01T10:13:08.9919374Z [command]/usr/bin/git rev-parse --verify --quiet c13d400bffe90e16b96520bbc8a41a6f0c9cd584^{object}
2022-12-01T10:13:08.9959380Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:13:10.2798109Z ##[endgroup]
2022-12-01T10:13:10.2798854Z ##[group]Determining the checkout info
2022-12-01T10:13:10.2800144Z ##[endgroup]
2022-12-01T10:13:10.2800568Z ##[group]Checking out the ref
2022-12-01T10:13:10.2805452Z [command]/usr/bin/git checkout --quiet --force c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:13:11.9280795Z ##[endgroup]
2022-12-01T10:13:11.9281617Z ##[group]Setting up auth for fetching submodules
2022-12-01T10:13:11.9288096Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-12-01T10:13:11.9338952Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-12-01T10:13:11.9372083Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-12-01T10:13:11.9404102Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-12-01T10:13:11.9432242Z ##[endgroup]
2022-12-01T10:13:11.9433056Z ##[group]Fetching submodules
2022-12-01T10:13:11.9437014Z [command]/usr/bin/git submodule sync --recursive
2022-12-01T10:13:11.9760417Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-12-01T10:13:12.0070505Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-12-01T10:13:12.0072799Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-12-01T10:13:12.0075961Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-12-01T10:13:12.0079356Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-12-01T10:13:12.0082850Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-12-01T10:13:12.0086843Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-12-01T10:13:12.0090371Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-12-01T10:13:12.0094325Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-12-01T10:13:12.0098944Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-12-01T10:13:12.0103043Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-12-01T10:13:12.0107556Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-12-01T10:13:12.0111869Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-12-01T10:13:12.0116439Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-12-01T10:13:12.0121179Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-12-01T10:13:12.0126050Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-12-01T10:13:12.0130889Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-12-01T10:13:12.0136336Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-12-01T10:13:12.0141634Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:13:12.0146927Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-12-01T10:13:12.0152638Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-12-01T10:13:12.0158059Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-12-01T10:13:12.0164071Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-12-01T10:13:12.0170044Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-12-01T10:13:12.0176431Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-12-01T10:13:12.0182534Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-12-01T10:13:12.0188864Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-12-01T10:13:12.0195238Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-12-01T10:13:12.0201676Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-12-01T10:13:12.0208394Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-12-01T10:13:12.0215661Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-12-01T10:13:12.0222598Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-12-01T10:13:12.0229544Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-12-01T10:13:12.0236702Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-12-01T10:13:12.0244076Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-12-01T10:13:12.0251482Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-12-01T10:13:12.0259352Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-12-01T10:13:12.0266862Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-12-01T10:13:12.0274599Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-12-01T10:13:12.0282579Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-12-01T10:13:12.0290756Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-12-01T10:13:12.0299491Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-12-01T10:13:12.0327899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-12-01T10:13:12.3158150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-12-01T10:13:12.5454141Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-12-01T10:13:12.7634303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-12-01T10:13:13.0737147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-12-01T10:13:13.3767722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-12-01T10:13:15.4294855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-12-01T10:13:22.6096210Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-12-01T10:13:23.0199378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-12-01T10:13:23.6239990Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-12-01T10:13:25.2220383Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-12-01T10:13:26.5412187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-12-01T10:13:28.1565649Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-12-01T10:13:34.9501980Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-12-01T10:13:35.7776833Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-12-01T10:13:37.0610812Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-12-01T10:13:38.2219399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-12-01T10:13:38.4719741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-12-01T10:13:39.0067617Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-12-01T10:13:39.4839496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-12-01T10:13:40.5122794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-12-01T10:13:40.9837854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-12-01T10:13:41.2330043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-12-01T10:13:41.5058491Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-12-01T10:13:44.1911282Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-12-01T10:13:44.6429839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-12-01T10:13:45.0350487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-12-01T10:13:51.1713673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-12-01T10:13:52.7931997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-12-01T10:13:53.2354720Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-12-01T10:13:53.4566482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-12-01T10:13:59.3433232Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-12-01T10:13:59.7853994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-12-01T10:14:00.0354871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-12-01T10:14:01.0332320Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-12-01T10:14:01.4445513Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-12-01T10:14:02.0510046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-12-01T10:14:02.5198114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-12-01T10:14:03.2264938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-12-01T10:14:05.9410111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-12-01T10:14:06.5058238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-12-01T10:14:08.7644868Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-12-01T10:14:08.7769248Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-12-01T10:14:08.7865957Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-12-01T10:14:08.8144312Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-12-01T10:14:08.8411098Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-12-01T10:14:08.8863085Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-12-01T10:14:09.6555889Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-12-01T10:14:09.6810187Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-12-01T10:14:09.8031292Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-12-01T10:14:09.8438139Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-12-01T10:14:10.1975867Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-12-01T10:14:10.7163379Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-12-01T10:14:11.0135407Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-12-01T10:14:11.0700864Z Submodule path 'third_party/fbgemm': checked out '0d98c261561524cce92e37fe307ea6596664309a'
2022-12-01T10:14:11.0717750Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:11.0720816Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:11.0724121Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:11.0727644Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:11.0754865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-12-01T10:14:12.0419737Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-12-01T10:14:12.6320511Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-12-01T10:14:13.6451697Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-12-01T10:14:14.0615133Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-12-01T10:14:14.1843420Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-12-01T10:14:14.2554554Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-12-01T10:14:14.2669885Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-12-01T10:14:14.3706889Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-12-01T10:14:14.4139832Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-12-01T10:14:14.4240562Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-12-01T10:14:14.4710385Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-12-01T10:14:14.5000978Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72'
2022-12-01T10:14:14.5552967Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-12-01T10:14:14.5684811Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99'
2022-12-01T10:14:14.5702485Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:14.5729313Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-12-01T10:14:22.8011329Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-12-01T10:14:22.8030528Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:22.8059272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-12-01T10:14:31.1169690Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-12-01T10:14:31.1288344Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-12-01T10:14:31.1461182Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-12-01T10:14:31.2613986Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573'
2022-12-01T10:14:31.2632134Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:31.2635274Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:31.2662746Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-12-01T10:14:32.4273074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-12-01T10:14:33.5349495Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-12-01T10:14:33.5996889Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-12-01T10:14:33.6236199Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-12-01T10:14:33.6391492Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-12-01T10:14:33.7695428Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-12-01T10:14:34.0873544Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-12-01T10:14:34.0906745Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:34.0910052Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:34.0939884Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-12-01T10:14:34.5247164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-12-01T10:14:35.3947169Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-12-01T10:14:35.4330757Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-12-01T10:14:35.4511735Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-12-01T10:14:35.4529136Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:35.4556071Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-12-01T10:14:37.2131357Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-12-01T10:14:37.2154994Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:37.2159018Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:37.2186811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-12-01T10:14:37.6576538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-12-01T10:14:38.5706064Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-12-01T10:14:38.6493869Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-12-01T10:14:38.6503043Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:38.6529848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-12-01T10:14:38.9040419Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-12-01T10:14:38.9145112Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-12-01T10:14:39.2356360Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-12-01T10:14:39.2379336Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:39.2382497Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:39.2409679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-12-01T10:14:39.6505558Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-12-01T10:14:41.7205309Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-12-01T10:14:41.8033396Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-12-01T10:14:41.8129708Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-12-01T10:14:41.8305669Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-12-01T10:14:41.8660217Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020'
2022-12-01T10:14:41.8760813Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-12-01T10:14:41.9091551Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-12-01T10:14:41.9196234Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-12-01T10:14:41.9715118Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-12-01T10:14:42.1084148Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-12-01T10:14:42.1400807Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-12-01T10:14:42.1418961Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:42.1422191Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:42.1425559Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:42.1429028Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:42.1457116Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-12-01T10:14:43.1850149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-12-01T10:14:43.4924778Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-12-01T10:14:44.8707558Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-12-01T10:14:45.7573816Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-12-01T10:14:45.7745410Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-12-01T10:14:45.8523667Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-12-01T10:14:45.8850177Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-12-01T10:14:45.8867979Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:45.8894714Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-12-01T10:14:46.1135690Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-12-01T10:14:46.2735510Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-12-01T10:14:46.2767995Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-12-01T10:14:46.3093368Z Entering 'android/libs/fbjni'
2022-12-01T10:14:46.3136080Z Entering 'third_party/FP16'
2022-12-01T10:14:46.3180347Z Entering 'third_party/FXdiv'
2022-12-01T10:14:46.3223586Z Entering 'third_party/NNPACK'
2022-12-01T10:14:46.3266413Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:46.3309273Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:46.3351644Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:46.3405924Z Entering 'third_party/benchmark'
2022-12-01T10:14:46.3449489Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:46.3492323Z Entering 'third_party/cub'
2022-12-01T10:14:46.3535395Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:46.3582487Z Entering 'third_party/cutlass'
2022-12-01T10:14:46.3632179Z Entering 'third_party/eigen'
2022-12-01T10:14:46.3677174Z Entering 'third_party/fbgemm'
2022-12-01T10:14:46.3720570Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:46.3762047Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:46.3804809Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:46.3846756Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:46.3890977Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:46.3935430Z Entering 'third_party/fmt'
2022-12-01T10:14:46.3976534Z Entering 'third_party/foxi'
2022-12-01T10:14:46.4019040Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:46.4060527Z Entering 'third_party/gloo'
2022-12-01T10:14:46.4103303Z Entering 'third_party/googletest'
2022-12-01T10:14:46.4146614Z Entering 'third_party/ideep'
2022-12-01T10:14:46.4189254Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:46.4233309Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:46.4283974Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:46.4326048Z Entering 'third_party/ittapi'
2022-12-01T10:14:46.4369025Z Entering 'third_party/kineto'
2022-12-01T10:14:46.4411342Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:46.4454082Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:46.4498188Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:46.4540343Z Entering 'third_party/neon2sse'
2022-12-01T10:14:46.4582336Z Entering 'third_party/nlohmann'
2022-12-01T10:14:46.4625993Z Entering 'third_party/onnx'
2022-12-01T10:14:46.4681887Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:46.4724832Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:46.4769426Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:46.4810474Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:46.4858111Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:46.4901384Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:46.4943724Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:46.4991740Z Entering 'third_party/pocketfft'
2022-12-01T10:14:46.5034563Z Entering 'third_party/protobuf'
2022-12-01T10:14:46.5079798Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:46.5121426Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:46.5165371Z Entering 'third_party/psimd'
2022-12-01T10:14:46.5207692Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:46.5249574Z Entering 'third_party/pybind11'
2022-12-01T10:14:46.5292212Z Entering 'third_party/python-enum'
2022-12-01T10:14:46.5334822Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:46.5376200Z Entering 'third_party/python-six'
2022-12-01T10:14:46.5417649Z Entering 'third_party/sleef'
2022-12-01T10:14:46.5459279Z Entering 'third_party/tbb'
2022-12-01T10:14:46.5503005Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:46.5545107Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:46.5589121Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:46.5630480Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:46.5674420Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:46.5716306Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:46.5762674Z Entering 'third_party/zstd'
2022-12-01T10:14:46.5813869Z ##[endgroup]
2022-12-01T10:14:46.5816227Z ##[group]Persisting credentials for submodules
2022-12-01T10:14:46.5822756Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-12-01T10:14:46.6131698Z Entering 'android/libs/fbjni'
2022-12-01T10:14:46.6173434Z Entering 'third_party/FP16'
2022-12-01T10:14:46.6215298Z Entering 'third_party/FXdiv'
2022-12-01T10:14:46.6256699Z Entering 'third_party/NNPACK'
2022-12-01T10:14:46.6298489Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:46.6340843Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:46.6382782Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:46.6435224Z Entering 'third_party/benchmark'
2022-12-01T10:14:46.6476344Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:46.6519120Z Entering 'third_party/cub'
2022-12-01T10:14:46.6560941Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:46.6608969Z Entering 'third_party/cutlass'
2022-12-01T10:14:46.6658872Z Entering 'third_party/eigen'
2022-12-01T10:14:46.6703342Z Entering 'third_party/fbgemm'
2022-12-01T10:14:46.6744331Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:46.6786072Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:46.6827232Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:46.6868110Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:46.6910238Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:46.6954881Z Entering 'third_party/fmt'
2022-12-01T10:14:46.6996468Z Entering 'third_party/foxi'
2022-12-01T10:14:46.7037739Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:46.7079429Z Entering 'third_party/gloo'
2022-12-01T10:14:46.7121860Z Entering 'third_party/googletest'
2022-12-01T10:14:46.7163573Z Entering 'third_party/ideep'
2022-12-01T10:14:46.7204820Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:46.7248084Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:46.7296913Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:46.7338723Z Entering 'third_party/ittapi'
2022-12-01T10:14:46.7382143Z Entering 'third_party/kineto'
2022-12-01T10:14:46.7423275Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:46.7464507Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:46.7508234Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:46.7549253Z Entering 'third_party/neon2sse'
2022-12-01T10:14:46.7590962Z Entering 'third_party/nlohmann'
2022-12-01T10:14:46.7635072Z Entering 'third_party/onnx'
2022-12-01T10:14:46.7689764Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:46.7731403Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:46.7774995Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:46.7818510Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:46.7865476Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:46.7907917Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:46.7949493Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:46.7995546Z Entering 'third_party/pocketfft'
2022-12-01T10:14:46.8037411Z Entering 'third_party/protobuf'
2022-12-01T10:14:46.8083023Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:46.8125238Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:46.8168703Z Entering 'third_party/psimd'
2022-12-01T10:14:46.8210703Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:46.8252455Z Entering 'third_party/pybind11'
2022-12-01T10:14:46.8295668Z Entering 'third_party/python-enum'
2022-12-01T10:14:46.8336758Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:46.8378496Z Entering 'third_party/python-six'
2022-12-01T10:14:46.8419934Z Entering 'third_party/sleef'
2022-12-01T10:14:46.8461481Z Entering 'third_party/tbb'
2022-12-01T10:14:46.8505086Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:46.8547316Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:46.8590335Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:46.8632771Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:46.8675143Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:46.8716543Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:46.8761245Z Entering 'third_party/zstd'
2022-12-01T10:14:46.8816849Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-12-01T10:14:46.9130781Z Entering 'android/libs/fbjni'
2022-12-01T10:14:46.9171641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-12-01T10:14:46.9188711Z Entering 'third_party/FP16'
2022-12-01T10:14:46.9228297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-12-01T10:14:46.9245238Z Entering 'third_party/FXdiv'
2022-12-01T10:14:46.9286037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-12-01T10:14:46.9303657Z Entering 'third_party/NNPACK'
2022-12-01T10:14:46.9343531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-12-01T10:14:46.9361410Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:46.9400468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-12-01T10:14:46.9419412Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:46.9458270Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-12-01T10:14:46.9475459Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:46.9514565Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-12-01T10:14:46.9544137Z Entering 'third_party/benchmark'
2022-12-01T10:14:46.9583443Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:46.9600521Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:46.9639225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-12-01T10:14:46.9657627Z Entering 'third_party/cub'
2022-12-01T10:14:46.9697191Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-12-01T10:14:46.9714723Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:46.9752898Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-12-01T10:14:46.9775592Z Entering 'third_party/cutlass'
2022-12-01T10:14:46.9813745Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-12-01T10:14:46.9838796Z Entering 'third_party/eigen'
2022-12-01T10:14:46.9879465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-12-01T10:14:46.9898946Z Entering 'third_party/fbgemm'
2022-12-01T10:14:46.9938633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-12-01T10:14:46.9956094Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:46.9995207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-12-01T10:14:47.0012382Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:47.0052604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-12-01T10:14:47.0070448Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:47.0108631Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:47.0125570Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:47.0166499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-12-01T10:14:47.0184959Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:47.0225622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-12-01T10:14:47.0244475Z Entering 'third_party/fmt'
2022-12-01T10:14:47.0283533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-12-01T10:14:47.0301841Z Entering 'third_party/foxi'
2022-12-01T10:14:47.0341139Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-12-01T10:14:47.0358015Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:47.0397417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-12-01T10:14:47.0414581Z Entering 'third_party/gloo'
2022-12-01T10:14:47.0452874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-12-01T10:14:47.0470328Z Entering 'third_party/googletest'
2022-12-01T10:14:47.0510401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:47.0527920Z Entering 'third_party/ideep'
2022-12-01T10:14:47.0566551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-12-01T10:14:47.0583244Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:47.0621183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-12-01T10:14:47.0639810Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:47.0679513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-12-01T10:14:47.0703614Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:47.0742429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-12-01T10:14:47.0759263Z Entering 'third_party/ittapi'
2022-12-01T10:14:47.0799025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-12-01T10:14:47.0816212Z Entering 'third_party/kineto'
2022-12-01T10:14:47.0854061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-12-01T10:14:47.0871466Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:47.0911765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-12-01T10:14:47.0928788Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:47.0968120Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:47.0987942Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:47.1026698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-12-01T10:14:47.1043789Z Entering 'third_party/neon2sse'
2022-12-01T10:14:47.1082509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-12-01T10:14:47.1100091Z Entering 'third_party/nlohmann'
2022-12-01T10:14:47.1139425Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-12-01T10:14:47.1158171Z Entering 'third_party/onnx'
2022-12-01T10:14:47.1197141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-12-01T10:14:47.1227448Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:47.1266284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:47.1283377Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:47.1322174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:47.1341617Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:47.1380604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-12-01T10:14:47.1396919Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:47.1436082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-12-01T10:14:47.1459788Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:47.1499802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:47.1517499Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:47.1557936Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:47.1574760Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:47.1614113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-12-01T10:14:47.1636007Z Entering 'third_party/pocketfft'
2022-12-01T10:14:47.1674742Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-12-01T10:14:47.1691834Z Entering 'third_party/protobuf'
2022-12-01T10:14:47.1732035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-12-01T10:14:47.1753033Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:47.1791436Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:47.1809029Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:47.1847934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:47.1867723Z Entering 'third_party/psimd'
2022-12-01T10:14:47.1907173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-12-01T10:14:47.1924281Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:47.1964223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-12-01T10:14:47.1981672Z Entering 'third_party/pybind11'
2022-12-01T10:14:47.2019829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:47.2037108Z Entering 'third_party/python-enum'
2022-12-01T10:14:47.2077635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-12-01T10:14:47.2094553Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:47.2132390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-12-01T10:14:47.2149567Z Entering 'third_party/python-six'
2022-12-01T10:14:47.2187922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-12-01T10:14:47.2205025Z Entering 'third_party/sleef'
2022-12-01T10:14:47.2244250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-12-01T10:14:47.2261859Z Entering 'third_party/tbb'
2022-12-01T10:14:47.2300643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-12-01T10:14:47.2319794Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:47.2360194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-12-01T10:14:47.2377376Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:47.2415734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:47.2433281Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:47.2472099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-12-01T10:14:47.2488951Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:47.2527756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-12-01T10:14:47.2546698Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:47.2585209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:47.2601467Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:47.2641089Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-12-01T10:14:47.2661323Z Entering 'third_party/zstd'
2022-12-01T10:14:47.2700653Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-12-01T10:14:47.3485548Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-12-01T10:14:47.3798531Z Entering 'android/libs/fbjni'
2022-12-01T10:14:47.3841816Z Entering 'third_party/FP16'
2022-12-01T10:14:47.3883976Z Entering 'third_party/FXdiv'
2022-12-01T10:14:47.3926177Z Entering 'third_party/NNPACK'
2022-12-01T10:14:47.3968960Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:47.4011094Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:47.4053503Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:47.4107298Z Entering 'third_party/benchmark'
2022-12-01T10:14:47.4149324Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:47.4193393Z Entering 'third_party/cub'
2022-12-01T10:14:47.4235524Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:47.4283483Z Entering 'third_party/cutlass'
2022-12-01T10:14:47.4331977Z Entering 'third_party/eigen'
2022-12-01T10:14:47.4377506Z Entering 'third_party/fbgemm'
2022-12-01T10:14:47.4419735Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:47.4463149Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:47.4504824Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:47.4546160Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:47.4589236Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:47.4635676Z Entering 'third_party/fmt'
2022-12-01T10:14:47.4678698Z Entering 'third_party/foxi'
2022-12-01T10:14:47.4721310Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:47.4765147Z Entering 'third_party/gloo'
2022-12-01T10:14:47.4807792Z Entering 'third_party/googletest'
2022-12-01T10:14:47.4850895Z Entering 'third_party/ideep'
2022-12-01T10:14:47.4892918Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:47.4938403Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:47.4986417Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:47.5028789Z Entering 'third_party/ittapi'
2022-12-01T10:14:47.5070884Z Entering 'third_party/kineto'
2022-12-01T10:14:47.5114137Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:47.5155926Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:47.5209265Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:47.5252791Z Entering 'third_party/neon2sse'
2022-12-01T10:14:47.5297455Z Entering 'third_party/nlohmann'
2022-12-01T10:14:47.5342009Z Entering 'third_party/onnx'
2022-12-01T10:14:47.5397971Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:47.5442741Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:47.5488045Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:47.5531931Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:47.5581793Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:47.5625568Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:47.5668224Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:47.5716684Z Entering 'third_party/pocketfft'
2022-12-01T10:14:47.5761048Z Entering 'third_party/protobuf'
2022-12-01T10:14:47.5808522Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:47.5853079Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:47.5899947Z Entering 'third_party/psimd'
2022-12-01T10:14:47.5943873Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:47.5986934Z Entering 'third_party/pybind11'
2022-12-01T10:14:47.6030025Z Entering 'third_party/python-enum'
2022-12-01T10:14:47.6072713Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:47.6115740Z Entering 'third_party/python-six'
2022-12-01T10:14:47.6158480Z Entering 'third_party/sleef'
2022-12-01T10:14:47.6203597Z Entering 'third_party/tbb'
2022-12-01T10:14:47.6250729Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:47.6294346Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:47.6337232Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:47.6380222Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:47.6424150Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:47.6465991Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:47.6512732Z Entering 'third_party/zstd'
2022-12-01T10:14:47.6568683Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-12-01T10:14:47.6883800Z Entering 'android/libs/fbjni'
2022-12-01T10:14:47.6926524Z Entering 'third_party/FP16'
2022-12-01T10:14:47.6969226Z Entering 'third_party/FXdiv'
2022-12-01T10:14:47.7012161Z Entering 'third_party/NNPACK'
2022-12-01T10:14:47.7056583Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:47.7099329Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:47.7143301Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:47.7197145Z Entering 'third_party/benchmark'
2022-12-01T10:14:47.7240129Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:47.7283436Z Entering 'third_party/cub'
2022-12-01T10:14:47.7328083Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:47.7376246Z Entering 'third_party/cutlass'
2022-12-01T10:14:47.7425707Z Entering 'third_party/eigen'
2022-12-01T10:14:47.7471066Z Entering 'third_party/fbgemm'
2022-12-01T10:14:47.7514450Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:47.7556866Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:47.7600522Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:47.7643437Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:47.7686199Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:47.7731187Z Entering 'third_party/fmt'
2022-12-01T10:14:47.7774359Z Entering 'third_party/foxi'
2022-12-01T10:14:47.7818466Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:47.7860822Z Entering 'third_party/gloo'
2022-12-01T10:14:47.7905001Z Entering 'third_party/googletest'
2022-12-01T10:14:47.7947964Z Entering 'third_party/ideep'
2022-12-01T10:14:47.7989680Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:47.8034066Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:47.8083757Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:47.8126190Z Entering 'third_party/ittapi'
2022-12-01T10:14:47.8169245Z Entering 'third_party/kineto'
2022-12-01T10:14:47.8212407Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:47.8255201Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:47.8299638Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:47.8342738Z Entering 'third_party/neon2sse'
2022-12-01T10:14:47.8385368Z Entering 'third_party/nlohmann'
2022-12-01T10:14:47.8429081Z Entering 'third_party/onnx'
2022-12-01T10:14:47.8486690Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:47.8530081Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:47.8575228Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:47.8617353Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:47.8665123Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:47.8708533Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:47.8750612Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:47.8797922Z Entering 'third_party/pocketfft'
2022-12-01T10:14:47.8841743Z Entering 'third_party/protobuf'
2022-12-01T10:14:47.8888470Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:47.8930956Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:47.8975562Z Entering 'third_party/psimd'
2022-12-01T10:14:47.9019488Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:47.9061912Z Entering 'third_party/pybind11'
2022-12-01T10:14:47.9105829Z Entering 'third_party/python-enum'
2022-12-01T10:14:47.9148125Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:47.9190876Z Entering 'third_party/python-six'
2022-12-01T10:14:47.9233558Z Entering 'third_party/sleef'
2022-12-01T10:14:47.9276884Z Entering 'third_party/tbb'
2022-12-01T10:14:47.9321633Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:47.9364031Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:47.9405788Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:47.9447402Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:47.9489591Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:47.9530602Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:47.9575990Z Entering 'third_party/zstd'
2022-12-01T10:14:47.9628873Z ##[endgroup]
2022-12-01T10:14:47.9672646Z [command]/usr/bin/git log -1 --format='%H'
2022-12-01T10:14:47.9703080Z 'c13d400bffe90e16b96520bbc8a41a6f0c9cd584'
2022-12-01T10:14:47.9864880Z Prepare all required actions
2022-12-01T10:14:47.9899069Z ##[group]Run ./.github/actions/setup-linux
2022-12-01T10:14:47.9899345Z env:
2022-12-01T10:14:47.9899580Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:47.9899818Z ##[endgroup]
2022-12-01T10:14:47.9919738Z ##[group]Run set -euo pipefail
2022-12-01T10:14:47.9920050Z set -euo pipefail
2022-12-01T10:14:47.9920338Z function get_ec2_metadata() {
2022-12-01T10:14:47.9920709Z   # Pulled from instance metadata endpoint for EC2
2022-12-01T10:14:47.9921137Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-12-01T10:14:47.9921539Z   category=$1
2022-12-01T10:14:47.9921852Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-12-01T10:14:47.9922296Z }
2022-12-01T10:14:47.9922575Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-12-01T10:14:47.9922945Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-12-01T10:14:47.9923326Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-12-01T10:14:47.9923666Z echo "system info $(uname -a)"
2022-12-01T10:14:47.9936972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:47.9937270Z env:
2022-12-01T10:14:47.9937508Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:47.9937750Z ##[endgroup]
2022-12-01T10:14:48.0042105Z ami-id: ami-096198a0bccc6bad4
2022-12-01T10:14:48.0106066Z instance-id: i-0513dad604108ca34
2022-12-01T10:14:48.0167014Z instance-type: g3.8xlarge
2022-12-01T10:14:48.0175927Z system info Linux ip-10-0-2-60.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-12-01T10:14:48.0196953Z ##[group]Run if systemctl is-active --quiet docker; then
2022-12-01T10:14:48.0197336Z if systemctl is-active --quiet docker; then
2022-12-01T10:14:48.0197652Z     echo "Docker daemon is running...";
2022-12-01T10:14:48.0197936Z else
2022-12-01T10:14:48.0198257Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-12-01T10:14:48.0198564Z fi
2022-12-01T10:14:48.0209843Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:48.0210138Z env:
2022-12-01T10:14:48.0210378Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:48.0210619Z ##[endgroup]
2022-12-01T10:14:48.0261833Z Docker daemon is running...
2022-12-01T10:14:48.0282496Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-12-01T10:14:48.0282967Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-12-01T10:14:48.0283335Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:14:48.0283837Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-12-01T10:14:48.0284308Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-12-01T10:14:48.0296200Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:48.0296497Z env:
2022-12-01T10:14:48.0296716Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:48.0296981Z   AWS_RETRY_MODE: standard
2022-12-01T10:14:48.0297236Z   AWS_MAX_ATTEMPTS: 5
2022-12-01T10:14:48.0297483Z   AWS_DEFAULT_REGION: us-east-1
2022-12-01T10:14:48.0297739Z ##[endgroup]
2022-12-01T10:14:48.9616295Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-12-01T10:14:48.9616764Z Configure a credential helper to remove this warning. See
2022-12-01T10:14:48.9617301Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-12-01T10:14:48.9617570Z 
2022-12-01T10:14:48.9618489Z Login Succeeded
2022-12-01T10:14:48.9723924Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:14:48.9724336Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:14:48.9724844Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:14:48.9737651Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:48.9737954Z env:
2022-12-01T10:14:48.9738195Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:48.9738436Z ##[endgroup]
2022-12-01T10:14:48.9830722Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-12-01T10:14:48.9831068Z with:
2022-12-01T10:14:48.9831485Z   github-secret: ***
2022-12-01T10:14:48.9831764Z   activate-with-label: false
2022-12-01T10:14:48.9832033Z   label: with-ssh
2022-12-01T10:14:48.9832277Z   remove-existing-keys: true
2022-12-01T10:14:48.9832528Z env:
2022-12-01T10:14:48.9832764Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:48.9833004Z ##[endgroup]
2022-12-01T10:14:49.4186954Z Grabbing public ssh keys from https://github.com/charlie-wt.keys
2022-12-01T10:14:49.5111657Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2022-12-01T10:14:49.5133290Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2022-12-01T10:14:49.5180729Z Login using: ssh ec2-user@ec2-3-85-50-8.compute-1.amazonaws.com
2022-12-01T10:14:49.5261949Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-12-01T10:14:49.5262315Z with:
2022-12-01T10:14:49.5262776Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:14:49.5263240Z env:
2022-12-01T10:14:49.5263483Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:49.5263721Z ##[endgroup]
2022-12-01T10:14:49.5281227Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:14:49.5281608Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:14:49.5281979Z # ignore output since only exit code is used for conditional
2022-12-01T10:14:49.5282358Z # only pull docker image if it's not available locally
2022-12-01T10:14:49.5282742Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-12-01T10:14:49.5283141Z   retry docker pull "${DOCKER_IMAGE}"
2022-12-01T10:14:49.5283415Z fi
2022-12-01T10:14:49.5296270Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:49.5296565Z env:
2022-12-01T10:14:49.5296801Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:49.5297292Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:14:49.5297765Z ##[endgroup]
2022-12-01T10:14:49.7761902Z fa72f5a0a230eb632055220542038bd4ceca184b: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-12-01T10:14:49.7762725Z 40dd5be53814: Pulling fs layer
2022-12-01T10:14:49.7763200Z bd44602516a4: Pulling fs layer
2022-12-01T10:14:49.7763484Z 8ebfb31ea67d: Pulling fs layer
2022-12-01T10:14:49.7763678Z 1589dc294916: Pulling fs layer
2022-12-01T10:14:49.7764290Z 2c3a764ff1ef: Pulling fs layer
2022-12-01T10:14:49.7764746Z 2fb24fb5f7cb: Pulling fs layer
2022-12-01T10:14:49.7765056Z d6e4b45751c9: Pulling fs layer
2022-12-01T10:14:49.7765381Z 98a26bc0781e: Pulling fs layer
2022-12-01T10:14:49.7765596Z 07c42b0591b2: Pulling fs layer
2022-12-01T10:14:49.7766696Z a7b4d9098b01: Pulling fs layer
2022-12-01T10:14:49.7766908Z 9b965084d7f3: Pulling fs layer
2022-12-01T10:14:49.7767279Z 306938abc720: Pulling fs layer
2022-12-01T10:14:49.7767712Z 06e616940749: Pulling fs layer
2022-12-01T10:14:49.7768010Z 540637333efa: Pulling fs layer
2022-12-01T10:14:49.7768218Z 5cd3ebaf3e1b: Pulling fs layer
2022-12-01T10:14:49.7768491Z dd7a31bfac90: Pulling fs layer
2022-12-01T10:14:49.7768742Z 6416ca3f405b: Pulling fs layer
2022-12-01T10:14:49.7769004Z 19c44dd1f4cf: Pulling fs layer
2022-12-01T10:14:49.7769279Z 33688bb73038: Pulling fs layer
2022-12-01T10:14:49.7770720Z 570b8809d75f: Pulling fs layer
2022-12-01T10:14:49.7771064Z 3ea8b8f31abf: Pulling fs layer
2022-12-01T10:14:49.7771352Z 690eecb8c64c: Pulling fs layer
2022-12-01T10:14:49.7771649Z e2285545a8bf: Pulling fs layer
2022-12-01T10:14:49.7771909Z 555faed4e286: Pulling fs layer
2022-12-01T10:14:49.7772175Z c53c37f056ac: Pulling fs layer
2022-12-01T10:14:49.7772430Z 75bd8ea35c95: Pulling fs layer
2022-12-01T10:14:49.7772682Z 2fb24fb5f7cb: Waiting
2022-12-01T10:14:49.7772911Z 9b965084d7f3: Waiting
2022-12-01T10:14:49.7773169Z b18c1d97e9da: Pulling fs layer
2022-12-01T10:14:49.7773425Z d90720873b3b: Pulling fs layer
2022-12-01T10:14:49.7773684Z 306938abc720: Waiting
2022-12-01T10:14:49.7773942Z 53b1597ec1c2: Pulling fs layer
2022-12-01T10:14:49.7774171Z 06e616940749: Waiting
2022-12-01T10:14:49.7774648Z ef984347a312: Pulling fs layer
2022-12-01T10:14:49.7774905Z d6e4b45751c9: Waiting
2022-12-01T10:14:49.7775175Z 343a1b6dc08c: Pulling fs layer
2022-12-01T10:14:49.7775620Z 16fdefc9f32c: Pulling fs layer
2022-12-01T10:14:49.7775873Z 540637333efa: Waiting
2022-12-01T10:14:49.7776060Z 690eecb8c64c: Waiting
2022-12-01T10:14:49.7776340Z dd7a31bfac90: Waiting
2022-12-01T10:14:49.7776561Z de0913f5235c: Pulling fs layer
2022-12-01T10:14:49.7776849Z e2285545a8bf: Waiting
2022-12-01T10:14:49.7777178Z 1589dc294916: Waiting
2022-12-01T10:14:49.7777416Z a7b4d9098b01: Waiting
2022-12-01T10:14:49.7777698Z bf99e81d1939: Pulling fs layer
2022-12-01T10:14:49.7778077Z c0f36b953c44: Pulling fs layer
2022-12-01T10:14:49.7778466Z 7456bfaa70b2: Pulling fs layer
2022-12-01T10:14:49.7778724Z 2c3a764ff1ef: Waiting
2022-12-01T10:14:49.7778957Z 5cd3ebaf3e1b: Waiting
2022-12-01T10:14:49.7779215Z 68e8ef95b145: Pulling fs layer
2022-12-01T10:14:49.7779537Z e497d45bb922: Pulling fs layer
2022-12-01T10:14:49.7779923Z 19c44dd1f4cf: Waiting
2022-12-01T10:14:49.7780221Z bf4a66dadd84: Pulling fs layer
2022-12-01T10:14:49.7780706Z 33688bb73038: Waiting
2022-12-01T10:14:49.7780956Z fd0d4928961f: Pulling fs layer
2022-12-01T10:14:49.7781194Z c53c37f056ac: Waiting
2022-12-01T10:14:49.7781422Z b18c1d97e9da: Waiting
2022-12-01T10:14:49.7781595Z 6df8346812f8: Pulling fs layer
2022-12-01T10:14:49.7781842Z 8b46f8aa8681: Pulling fs layer
2022-12-01T10:14:49.7782255Z 6db5688b5394: Pulling fs layer
2022-12-01T10:14:49.7782530Z daa053ac35f3: Pulling fs layer
2022-12-01T10:14:49.7782779Z c9a8257f8dc0: Pulling fs layer
2022-12-01T10:14:49.7783031Z 343a1b6dc08c: Waiting
2022-12-01T10:14:49.7783265Z de0913f5235c: Waiting
2022-12-01T10:14:49.7783492Z 30f21817172d: Pulling fs layer
2022-12-01T10:14:49.7783755Z bb9282afce05: Pulling fs layer
2022-12-01T10:14:49.7784019Z 909c475d21ec: Pulling fs layer
2022-12-01T10:14:49.7784251Z 7456bfaa70b2: Waiting
2022-12-01T10:14:49.7784487Z e497d45bb922: Waiting
2022-12-01T10:14:49.7784728Z 192b8944f15a: Pulling fs layer
2022-12-01T10:14:49.7784960Z 3ea8b8f31abf: Waiting
2022-12-01T10:14:49.7785209Z 3dd92c832839: Pulling fs layer
2022-12-01T10:14:49.7785473Z 5e1b0146a21d: Pulling fs layer
2022-12-01T10:14:49.7785718Z c76cf0339419: Pulling fs layer
2022-12-01T10:14:49.7785965Z 16fdefc9f32c: Waiting
2022-12-01T10:14:49.7786200Z 6df8346812f8: Waiting
2022-12-01T10:14:49.7786415Z 555faed4e286: Waiting
2022-12-01T10:14:49.7786668Z 2dc4928eecf0: Pulling fs layer
2022-12-01T10:14:49.7786924Z daa053ac35f3: Waiting
2022-12-01T10:14:49.7787145Z c9a8257f8dc0: Waiting
2022-12-01T10:14:49.7787377Z 30f21817172d: Waiting
2022-12-01T10:14:49.7787606Z 3dd92c832839: Waiting
2022-12-01T10:14:49.7787818Z 909c475d21ec: Waiting
2022-12-01T10:14:49.7788054Z 2dc4928eecf0: Waiting
2022-12-01T10:14:49.7788286Z c0f36b953c44: Waiting
2022-12-01T10:14:49.7788499Z 53b1597ec1c2: Waiting
2022-12-01T10:14:49.7788732Z 68e8ef95b145: Waiting
2022-12-01T10:14:49.7788962Z 192b8944f15a: Waiting
2022-12-01T10:14:49.7789172Z c76cf0339419: Waiting
2022-12-01T10:14:49.7789402Z 5e1b0146a21d: Waiting
2022-12-01T10:14:50.0551161Z bd44602516a4: Verifying Checksum
2022-12-01T10:14:50.0551570Z bd44602516a4: Download complete
2022-12-01T10:14:50.1566589Z 1589dc294916: Verifying Checksum
2022-12-01T10:14:50.2478743Z 2c3a764ff1ef: Download complete
2022-12-01T10:14:50.4332595Z 8ebfb31ea67d: Verifying Checksum
2022-12-01T10:14:50.4332941Z 8ebfb31ea67d: Download complete
2022-12-01T10:14:50.5271353Z d6e4b45751c9: Verifying Checksum
2022-12-01T10:14:50.5271717Z d6e4b45751c9: Download complete
2022-12-01T10:14:50.6924152Z 40dd5be53814: Verifying Checksum
2022-12-01T10:14:50.6924531Z 40dd5be53814: Download complete
2022-12-01T10:14:50.8150170Z 07c42b0591b2: Download complete
2022-12-01T10:14:50.9228161Z a7b4d9098b01: Download complete
2022-12-01T10:14:51.4496891Z 40dd5be53814: Pull complete
2022-12-01T10:14:51.7140208Z bd44602516a4: Pull complete
2022-12-01T10:14:52.2350656Z 8ebfb31ea67d: Pull complete
2022-12-01T10:14:52.3380322Z 1589dc294916: Pull complete
2022-12-01T10:14:52.4455573Z 2c3a764ff1ef: Pull complete
2022-12-01T10:14:57.4056227Z 9b965084d7f3: Verifying Checksum
2022-12-01T10:14:57.4056555Z 9b965084d7f3: Download complete
2022-12-01T10:14:57.4952220Z 306938abc720: Verifying Checksum
2022-12-01T10:14:57.4952558Z 306938abc720: Download complete
2022-12-01T10:14:57.5975311Z 06e616940749: Download complete
2022-12-01T10:14:57.7141173Z 540637333efa: Download complete
2022-12-01T10:15:00.0266864Z 5cd3ebaf3e1b: Verifying Checksum
2022-12-01T10:15:00.0267414Z 5cd3ebaf3e1b: Download complete
2022-12-01T10:15:00.1671180Z dd7a31bfac90: Verifying Checksum
2022-12-01T10:15:00.1671507Z dd7a31bfac90: Download complete
2022-12-01T10:15:00.2446049Z 6416ca3f405b: Verifying Checksum
2022-12-01T10:15:00.2446360Z 6416ca3f405b: Download complete
2022-12-01T10:15:24.2750239Z 2fb24fb5f7cb: Verifying Checksum
2022-12-01T10:15:24.2752643Z 2fb24fb5f7cb: Download complete
2022-12-01T10:15:24.3496604Z 33688bb73038: Download complete
2022-12-01T10:15:24.4597982Z 570b8809d75f: Verifying Checksum
2022-12-01T10:15:24.4598308Z 570b8809d75f: Download complete
2022-12-01T10:15:24.5586530Z 3ea8b8f31abf: Verifying Checksum
2022-12-01T10:15:24.5588919Z 3ea8b8f31abf: Download complete
2022-12-01T10:15:24.6625224Z 690eecb8c64c: Verifying Checksum
2022-12-01T10:15:24.6625596Z 690eecb8c64c: Download complete
2022-12-01T10:15:24.7543869Z e2285545a8bf: Verifying Checksum
2022-12-01T10:15:24.7544384Z e2285545a8bf: Download complete
2022-12-01T10:15:24.8378578Z 555faed4e286: Verifying Checksum
2022-12-01T10:15:24.8378966Z 555faed4e286: Download complete
2022-12-01T10:15:27.4161942Z c53c37f056ac: Verifying Checksum
2022-12-01T10:15:27.4162298Z c53c37f056ac: Download complete
2022-12-01T10:15:27.5087266Z 75bd8ea35c95: Verifying Checksum
2022-12-01T10:15:27.5087905Z 75bd8ea35c95: Download complete
2022-12-01T10:15:27.6092952Z b18c1d97e9da: Verifying Checksum
2022-12-01T10:15:27.6093543Z b18c1d97e9da: Download complete
2022-12-01T10:15:27.7107935Z d90720873b3b: Verifying Checksum
2022-12-01T10:15:27.7108400Z d90720873b3b: Download complete
2022-12-01T10:15:27.7926221Z 53b1597ec1c2: Download complete
2022-12-01T10:15:27.9221997Z ef984347a312: Verifying Checksum
2022-12-01T10:15:27.9222352Z ef984347a312: Download complete
2022-12-01T10:15:33.6622517Z 343a1b6dc08c: Verifying Checksum
2022-12-01T10:15:33.7413139Z 343a1b6dc08c: Download complete
2022-12-01T10:15:33.7413469Z 16fdefc9f32c: Verifying Checksum
2022-12-01T10:15:33.7413781Z 16fdefc9f32c: Download complete
2022-12-01T10:15:33.8447026Z de0913f5235c: Download complete
2022-12-01T10:15:33.9286876Z 98a26bc0781e: Verifying Checksum
2022-12-01T10:15:33.9287487Z 98a26bc0781e: Download complete
2022-12-01T10:15:34.0050384Z c0f36b953c44: Verifying Checksum
2022-12-01T10:15:34.0050768Z c0f36b953c44: Download complete
2022-12-01T10:15:34.0823483Z 7456bfaa70b2: Verifying Checksum
2022-12-01T10:15:34.0824026Z 7456bfaa70b2: Download complete
2022-12-01T10:15:34.6742870Z 68e8ef95b145: Verifying Checksum
2022-12-01T10:15:34.6743269Z 68e8ef95b145: Download complete
2022-12-01T10:15:34.7362459Z bf99e81d1939: Verifying Checksum
2022-12-01T10:15:34.7362941Z bf99e81d1939: Download complete
2022-12-01T10:15:34.7647870Z e497d45bb922: Verifying Checksum
2022-12-01T10:15:34.7648456Z e497d45bb922: Download complete
2022-12-01T10:15:34.8436859Z fd0d4928961f: Verifying Checksum
2022-12-01T10:15:34.8437145Z fd0d4928961f: Download complete
2022-12-01T10:15:34.9340117Z 6df8346812f8: Download complete
2022-12-01T10:15:35.9421250Z bf4a66dadd84: Verifying Checksum
2022-12-01T10:15:35.9421629Z bf4a66dadd84: Download complete
2022-12-01T10:15:36.0238338Z 6db5688b5394: Verifying Checksum
2022-12-01T10:15:36.0238700Z 6db5688b5394: Download complete
2022-12-01T10:15:36.1039787Z daa053ac35f3: Verifying Checksum
2022-12-01T10:15:36.1040142Z daa053ac35f3: Download complete
2022-12-01T10:15:36.1915271Z c9a8257f8dc0: Verifying Checksum
2022-12-01T10:15:36.1915866Z c9a8257f8dc0: Download complete
2022-12-01T10:15:36.3038202Z 30f21817172d: Verifying Checksum
2022-12-01T10:15:36.3038539Z 30f21817172d: Download complete
2022-12-01T10:15:36.7684879Z bb9282afce05: Verifying Checksum
2022-12-01T10:15:36.7685284Z bb9282afce05: Download complete
2022-12-01T10:15:36.8875909Z 909c475d21ec: Verifying Checksum
2022-12-01T10:15:36.8876266Z 909c475d21ec: Download complete
2022-12-01T10:15:38.5700940Z 192b8944f15a: Verifying Checksum
2022-12-01T10:15:38.5701288Z 192b8944f15a: Download complete
2022-12-01T10:15:38.6421183Z 3dd92c832839: Verifying Checksum
2022-12-01T10:15:38.6421474Z 3dd92c832839: Download complete
2022-12-01T10:15:39.7690609Z 2fb24fb5f7cb: Pull complete
2022-12-01T10:15:41.6164169Z d6e4b45751c9: Pull complete
2022-12-01T10:15:44.9519853Z 8b46f8aa8681: Verifying Checksum
2022-12-01T10:15:44.9520500Z 8b46f8aa8681: Download complete
2022-12-01T10:15:45.0407353Z c76cf0339419: Verifying Checksum
2022-12-01T10:15:45.0408029Z c76cf0339419: Download complete
2022-12-01T10:15:45.1337201Z 2dc4928eecf0: Download complete
2022-12-01T10:15:57.0504928Z 19c44dd1f4cf: Verifying Checksum
2022-12-01T10:15:57.0505528Z 19c44dd1f4cf: Download complete
2022-12-01T10:16:03.8886304Z 98a26bc0781e: Pull complete
2022-12-01T10:16:05.7676332Z 07c42b0591b2: Pull complete
2022-12-01T10:16:07.6139495Z a7b4d9098b01: Pull complete
2022-12-01T10:16:15.7934492Z 9b965084d7f3: Pull complete
2022-12-01T10:16:17.6711233Z 306938abc720: Pull complete
2022-12-01T10:16:19.8190417Z 06e616940749: Pull complete
2022-12-01T10:16:21.6980772Z 540637333efa: Pull complete
2022-12-01T10:16:25.9176902Z 5cd3ebaf3e1b: Pull complete
2022-12-01T10:16:27.6998786Z dd7a31bfac90: Pull complete
2022-12-01T10:16:29.5468641Z 6416ca3f405b: Pull complete
2022-12-01T10:17:03.6323483Z 19c44dd1f4cf: Pull complete
2022-12-01T10:17:05.5051387Z 33688bb73038: Pull complete
2022-12-01T10:17:07.0516943Z 5e1b0146a21d: Verifying Checksum
2022-12-01T10:17:07.0517384Z 5e1b0146a21d: Download complete
2022-12-01T10:17:07.3820159Z 570b8809d75f: Pull complete
2022-12-01T10:17:09.3429034Z 3ea8b8f31abf: Pull complete
2022-12-01T10:17:10.9754346Z 690eecb8c64c: Pull complete
2022-12-01T10:17:12.6464575Z e2285545a8bf: Pull complete
2022-12-01T10:17:14.5241144Z 555faed4e286: Pull complete
2022-12-01T10:17:18.6510414Z c53c37f056ac: Pull complete
2022-12-01T10:17:20.5301461Z 75bd8ea35c95: Pull complete
2022-12-01T10:17:22.6543533Z b18c1d97e9da: Pull complete
2022-12-01T10:17:25.3342211Z d90720873b3b: Pull complete
2022-12-01T10:17:27.9720927Z 53b1597ec1c2: Pull complete
2022-12-01T10:17:30.8900140Z ef984347a312: Pull complete
2022-12-01T10:17:39.6246528Z 343a1b6dc08c: Pull complete
2022-12-01T10:17:41.5224878Z 16fdefc9f32c: Pull complete
2022-12-01T10:17:43.4036746Z de0913f5235c: Pull complete
2022-12-01T10:17:45.9149428Z bf99e81d1939: Pull complete
2022-12-01T10:17:46.0259684Z c0f36b953c44: Pull complete
2022-12-01T10:17:46.1298518Z 7456bfaa70b2: Pull complete
2022-12-01T10:17:46.4905278Z 68e8ef95b145: Pull complete
2022-12-01T10:17:46.5901442Z e497d45bb922: Pull complete
2022-12-01T10:17:47.9343932Z bf4a66dadd84: Pull complete
2022-12-01T10:17:48.0344727Z fd0d4928961f: Pull complete
2022-12-01T10:17:48.1294324Z 6df8346812f8: Pull complete
2022-12-01T10:17:54.9620724Z 8b46f8aa8681: Pull complete
2022-12-01T10:17:55.5828843Z 6db5688b5394: Pull complete
2022-12-01T10:17:55.6870457Z daa053ac35f3: Pull complete
2022-12-01T10:17:55.7983793Z c9a8257f8dc0: Pull complete
2022-12-01T10:17:55.8939069Z 30f21817172d: Pull complete
2022-12-01T10:17:56.6984984Z bb9282afce05: Pull complete
2022-12-01T10:17:56.8028968Z 909c475d21ec: Pull complete
2022-12-01T10:17:58.7087920Z 192b8944f15a: Pull complete
2022-12-01T10:17:58.8018440Z 3dd92c832839: Pull complete
2022-12-01T10:18:41.2181511Z 5e1b0146a21d: Pull complete
2022-12-01T10:18:43.0879929Z c76cf0339419: Pull complete
2022-12-01T10:18:44.9991313Z 2dc4928eecf0: Pull complete
2022-12-01T10:18:46.3778571Z Digest: sha256:217fd7de680e1dd5bca4b2b4054bd05a8d454df5e210ffbf1e5955e01cf1f340
2022-12-01T10:18:46.8790366Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:18:47.1613551Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:18:47.1702772Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-12-01T10:18:47.1703124Z with:
2022-12-01T10:18:47.1703339Z   timeout_minutes: 10
2022-12-01T10:18:47.1703590Z   max_attempts: 3
2022-12-01T10:18:47.1703983Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-12-01T10:18:47.1704372Z   retry_wait_seconds: 10
2022-12-01T10:18:47.1704623Z   polling_interval_seconds: 1
2022-12-01T10:18:47.1704894Z   warning_on_retry: true
2022-12-01T10:18:47.1705154Z   continue_on_error: false
2022-12-01T10:18:47.1705378Z env:
2022-12-01T10:18:47.1705614Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:18:47.1705872Z ##[endgroup]
2022-12-01T10:18:47.2262911Z 
2022-12-01T10:18:47.2296888Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:18:47.2341826Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.57.run ==
2022-12-01T10:18:47.2344634Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-12-01T10:18:47.2345204Z + HAS_NVIDIA_DRIVER=0
2022-12-01T10:18:47.2346201Z ++ command -v nvidia-smi
2022-12-01T10:18:47.2347396Z + '[' -x '' ']'
2022-12-01T10:18:47.2347854Z + '[' 0 -eq 0 ']'
2022-12-01T10:18:47.2349787Z + sudo yum groupinstall -y 'Development Tools'
2022-12-01T10:18:47.7059566Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:18:48.0154959Z Resolving Dependencies
2022-12-01T10:18:48.0159563Z --> Running transaction check
2022-12-01T10:18:48.0162736Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-12-01T10:18:48.0375015Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-12-01T10:18:48.2447179Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-12-01T10:18:48.2448340Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-12-01T10:18:48.2493613Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-12-01T10:18:48.2501281Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-12-01T10:18:48.2512000Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-12-01T10:18:48.2626717Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-12-01T10:18:48.2633742Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-12-01T10:18:48.2679984Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2022-12-01T10:18:48.2703499Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-12-01T10:18:48.2712410Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-12-01T10:18:48.2720058Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-12-01T10:18:48.2821134Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-12-01T10:18:48.2953305Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-12-01T10:18:48.2971923Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.3141816Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3160782Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3214584Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3266774Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3320038Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3371317Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3423752Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3477671Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3638600Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3659434Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3679996Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.3706924Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.3740989Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2022-12-01T10:18:48.3801509Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-12-01T10:18:48.3815858Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-12-01T10:18:48.3866913Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-12-01T10:18:48.3881619Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-12-01T10:18:48.3900035Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-12-01T10:18:48.3928829Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-12-01T10:18:48.3965350Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-12-01T10:18:48.3990300Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-12-01T10:18:48.4022499Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-12-01T10:18:48.4258668Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-12-01T10:18:48.4275882Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-12-01T10:18:48.4299294Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-12-01T10:18:48.4466861Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4485762Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4486813Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4487829Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4488498Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4489154Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4489751Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4490399Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4491044Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4491665Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4492301Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4493003Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4493732Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4494583Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4495496Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4496403Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4514811Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4535106Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-12-01T10:18:48.4558960Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-12-01T10:18:48.4579777Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2022-12-01T10:18:48.4625834Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-12-01T10:18:48.4642515Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-12-01T10:18:48.4654672Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-12-01T10:18:48.4829965Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-12-01T10:18:48.4842650Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-12-01T10:18:48.4857837Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-12-01T10:18:48.4871131Z --> Running transaction check
2022-12-01T10:18:48.4874473Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-12-01T10:18:48.4954790Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-12-01T10:18:48.4992737Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-12-01T10:18:48.5006540Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5079135Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-12-01T10:18:48.5089909Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2022-12-01T10:18:48.5090974Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-12-01T10:18:48.5160302Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-12-01T10:18:48.5217838Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-12-01T10:18:48.5226447Z ---> Package glibc-devel.x86_64 0:2.26-62.amzn2 will be installed
2022-12-01T10:18:48.5346668Z --> Processing Dependency: glibc-headers = 2.26-62.amzn2 for package: glibc-devel-2.26-62.amzn2.x86_64
2022-12-01T10:18:48.5373595Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-62.amzn2.x86_64
2022-12-01T10:18:48.5374603Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2 will be installed
2022-12-01T10:18:48.5380308Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5393348Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5420060Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5454938Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5470779Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-12-01T10:18:48.5482891Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5498021Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5522810Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2022-12-01T10:18:48.5567787Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-12-01T10:18:48.5582962Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-12-01T10:18:48.5603395Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-12-01T10:18:48.5677182Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-12-01T10:18:48.5715230Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-12-01T10:18:48.5716514Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-12-01T10:18:48.5735027Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-12-01T10:18:48.5752151Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-12-01T10:18:48.5758275Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-12-01T10:18:48.5805558Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-12-01T10:18:48.5898917Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-12-01T10:18:48.5910694Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-12-01T10:18:48.5926316Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-12-01T10:18:48.5927518Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-12-01T10:18:48.5955633Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-12-01T10:18:48.6159570Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-12-01T10:18:48.6172695Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-12-01T10:18:48.6198665Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-12-01T10:18:48.6199481Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-12-01T10:18:48.6312463Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2022-12-01T10:18:48.7336431Z --> Running transaction check
2022-12-01T10:18:48.7337559Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-12-01T10:18:48.7346954Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-12-01T10:18:48.7372445Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-12-01T10:18:48.7373514Z ---> Package glibc-headers.x86_64 0:2.26-62.amzn2 will be installed
2022-12-01T10:18:48.7447464Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-62.amzn2.x86_64
2022-12-01T10:18:48.8547914Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-62.amzn2.x86_64
2022-12-01T10:18:48.8548741Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-12-01T10:18:48.8613828Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-12-01T10:18:48.8640699Z ---> Package kernel-devel.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-12-01T10:18:48.8667316Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.296-222.539.amzn2.x86_64
2022-12-01T10:18:48.8686399Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-12-01T10:18:48.8714252Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-12-01T10:18:48.8731704Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-12-01T10:18:48.8779656Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-12-01T10:18:48.8799574Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-12-01T10:18:48.8800687Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-12-01T10:18:48.8801510Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-12-01T10:18:48.8814817Z --> Running transaction check
2022-12-01T10:18:48.8816059Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-12-01T10:18:48.8833000Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-12-01T10:18:48.8845081Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-12-01T10:18:48.8872852Z ---> Package kernel-headers.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-12-01T10:18:48.8873844Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-12-01T10:18:48.8891231Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-12-01T10:18:48.8947924Z --> Running transaction check
2022-12-01T10:18:48.8948660Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2 will be installed
2022-12-01T10:18:49.1531318Z --> Finished Dependency Resolution
2022-12-01T10:18:49.2289282Z 
2022-12-01T10:18:49.2290009Z Dependencies Resolved
2022-12-01T10:18:49.2400398Z 
2022-12-01T10:18:49.2400703Z ================================================================================
2022-12-01T10:18:49.2401342Z  Package                 Arch     Version                    Repository    Size
2022-12-01T10:18:49.2401751Z ================================================================================
2022-12-01T10:18:49.2402152Z Installing for group install "Development Tools":
2022-12-01T10:18:49.2402854Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-12-01T10:18:49.2403718Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-12-01T10:18:49.2404374Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-12-01T10:18:49.2405345Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-12-01T10:18:49.2406035Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-12-01T10:18:49.2406501Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-12-01T10:18:49.2406927Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-12-01T10:18:49.2408129Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-12-01T10:18:49.2409089Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-12-01T10:18:49.2409974Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-12-01T10:18:49.2410476Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2022-12-01T10:18:49.2410898Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2022-12-01T10:18:49.2411327Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2022-12-01T10:18:49.2411775Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-12-01T10:18:49.2412489Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-12-01T10:18:49.2413321Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-12-01T10:18:49.2414237Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-12-01T10:18:49.2415370Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-12-01T10:18:49.2416291Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-12-01T10:18:49.2416895Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-12-01T10:18:49.2417819Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-12-01T10:18:49.2418458Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-12-01T10:18:49.2418889Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-12-01T10:18:49.2419338Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2022-12-01T10:18:49.2419775Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2022-12-01T10:18:49.2420088Z Installing for dependencies:
2022-12-01T10:18:49.2420497Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-12-01T10:18:49.2421013Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-12-01T10:18:49.2421566Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-12-01T10:18:49.2422130Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-12-01T10:18:49.2422570Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2022-12-01T10:18:49.2422957Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-12-01T10:18:49.2423390Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-12-01T10:18:49.2423852Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-12-01T10:18:49.2424306Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2022-12-01T10:18:49.2424730Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-12-01T10:18:49.2425174Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-12-01T10:18:49.2425633Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-12-01T10:18:49.2426066Z  glibc-devel             x86_64   2.26-62.amzn2              amzn2-core   995 k
2022-12-01T10:18:49.2426510Z  glibc-headers           x86_64   2.26-62.amzn2              amzn2-core   516 k
2022-12-01T10:18:49.2426942Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-12-01T10:18:49.2427385Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.2        amzn2-core    23 k
2022-12-01T10:18:49.2427824Z  kernel-devel            x86_64   4.14.296-222.539.amzn2     amzn2-core    13 M
2022-12-01T10:18:49.2428270Z  kernel-headers          x86_64   4.14.296-222.539.amzn2     amzn2-core   1.2 M
2022-12-01T10:18:49.2428711Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2022-12-01T10:18:49.2429124Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-12-01T10:18:49.2429552Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2022-12-01T10:18:49.2429980Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-12-01T10:18:49.2430418Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-12-01T10:18:49.2430831Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-12-01T10:18:49.2431248Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2022-12-01T10:18:49.2431676Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-12-01T10:18:49.2432092Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2022-12-01T10:18:49.2432526Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2022-12-01T10:18:49.2432946Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-12-01T10:18:49.2433363Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-12-01T10:18:49.2433770Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-12-01T10:18:49.2434195Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-12-01T10:18:49.2434621Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-12-01T10:18:49.2435050Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-12-01T10:18:49.2435509Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-12-01T10:18:49.2435971Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-12-01T10:18:49.2436444Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-12-01T10:18:49.2436892Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-12-01T10:18:49.2437354Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-12-01T10:18:49.2437803Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2022-12-01T10:18:49.2438373Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2022-12-01T10:18:49.2438822Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-12-01T10:18:49.2439249Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.2         amzn2-core    50 k
2022-12-01T10:18:49.2439458Z 
2022-12-01T10:18:49.2439575Z Transaction Summary
2022-12-01T10:18:49.2439844Z ================================================================================
2022-12-01T10:18:49.2440155Z Install  25 Packages (+43 Dependent packages)
2022-12-01T10:18:49.2440353Z 
2022-12-01T10:18:49.2440482Z Total download size: 96 M
2022-12-01T10:18:49.2440728Z Installed size: 303 M
2022-12-01T10:18:49.2440985Z Downloading packages:
2022-12-01T10:18:49.2453026Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-12-01T10:18:50.4632794Z --------------------------------------------------------------------------------
2022-12-01T10:18:50.4633321Z Total                                               79 MB/s |  96 MB  00:01     
2022-12-01T10:18:50.5688051Z Running transaction check
2022-12-01T10:18:50.6456128Z Running transaction test
2022-12-01T10:18:51.0608059Z Transaction test succeeded
2022-12-01T10:18:51.0611159Z Running transaction
2022-12-01T10:18:56.3129124Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2022-12-01T10:18:58.8001422Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2022-12-01T10:19:01.0351880Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2022-12-01T10:19:03.5302814Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2022-12-01T10:19:05.9826722Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2022-12-01T10:19:08.5162452Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2022-12-01T10:19:10.9700078Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2022-12-01T10:19:13.3296847Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2022-12-01T10:19:13.3485509Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2022-12-01T10:19:13.4285519Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2022-12-01T10:19:14.4853733Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2022-12-01T10:19:14.5184566Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2022-12-01T10:19:14.5490857Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         13/68 
2022-12-01T10:19:14.6049632Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        14/68 
2022-12-01T10:19:14.6606808Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   15/68 
2022-12-01T10:19:14.7661632Z   Installing : elfutils-0.176-2.amzn2.x86_64                              16/68 
2022-12-01T10:19:14.7941094Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         17/68 
2022-12-01T10:19:14.8158642Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 18/68 
2022-12-01T10:19:14.8455038Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          19/68 
2022-12-01T10:19:14.8696758Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        20/68 
2022-12-01T10:19:14.9442704Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               21/68 
2022-12-01T10:19:14.9906298Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      22/68 
2022-12-01T10:19:15.0726023Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2022-12-01T10:19:15.2484220Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2022-12-01T10:19:15.2818413Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2022-12-01T10:19:15.6720795Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               26/68 
2022-12-01T10:19:15.6960670Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              27/68 
2022-12-01T10:19:15.7187775Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               28/68 
2022-12-01T10:19:15.7368586Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      29/68 
2022-12-01T10:19:15.7651837Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  30/68 
2022-12-01T10:19:15.7891263Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               31/68 
2022-12-01T10:19:15.8779049Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  32/68 
2022-12-01T10:19:15.9605690Z   Installing : autoconf-2.69-11.amzn2.noarch                              33/68 
2022-12-01T10:19:16.0608851Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      34/68 
2022-12-01T10:19:16.0959285Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           35/68 
2022-12-01T10:19:16.1134124Z   Installing : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       36/68 
2022-12-01T10:19:16.1318341Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 37/68 
2022-12-01T10:19:16.4149995Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            38/68 
2022-12-01T10:19:16.5825233Z   Installing : kernel-headers-4.14.296-222.539.amzn2.x86_64               39/68 
2022-12-01T10:19:16.7125270Z   Installing : glibc-headers-2.26-62.amzn2.x86_64                         40/68 
2022-12-01T10:19:16.7463899Z   Installing : glibc-devel-2.26-62.amzn2.x86_64                           41/68 
2022-12-01T10:19:18.8078689Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2022-12-01T10:19:22.6153693Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2022-12-01T10:19:33.8988490Z   Installing : kernel-devel-4.14.296-222.539.amzn2.x86_64                 44/68 
2022-12-01T10:19:34.5061523Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2022-12-01T10:19:34.5553296Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2022-12-01T10:19:34.6087793Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2022-12-01T10:19:34.7402773Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2022-12-01T10:19:34.9161476Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2022-12-01T10:19:35.0174811Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-12-01T10:19:36.2328640Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2022-12-01T10:19:37.8466156Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2022-12-01T10:19:37.9615712Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2022-12-01T10:19:37.9997103Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2022-12-01T10:19:38.0359726Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2022-12-01T10:19:38.0921437Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2022-12-01T10:19:38.1538607Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2022-12-01T10:19:38.2562554Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2022-12-01T10:19:38.3184351Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2022-12-01T10:19:38.3627182Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2022-12-01T10:19:38.3981681Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2022-12-01T10:19:38.4409320Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2022-12-01T10:19:39.1176041Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2022-12-01T10:19:39.1572912Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2022-12-01T10:19:39.1832282Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2022-12-01T10:19:39.4927453Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2022-12-01T10:19:39.5322243Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2022-12-01T10:19:39.5965315Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2022-12-01T10:19:39.6096386Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2022-12-01T10:19:39.6228802Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2022-12-01T10:19:39.6354914Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2022-12-01T10:19:39.6466197Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2022-12-01T10:19:39.6608841Z   Verifying  : kernel-devel-4.14.296-222.539.amzn2.x86_64                  5/68 
2022-12-01T10:19:39.6723693Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              6/68 
2022-12-01T10:19:39.6816763Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            7/68 
2022-12-01T10:19:39.6917838Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         8/68 
2022-12-01T10:19:39.7003641Z   Verifying  : glibc-devel-2.26-62.amzn2.x86_64                            9/68 
2022-12-01T10:19:39.7083550Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                         10/68 
2022-12-01T10:19:39.7182765Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            11/68 
2022-12-01T10:19:39.7280005Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      12/68 
2022-12-01T10:19:39.7370987Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            13/68 
2022-12-01T10:19:39.7455504Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           14/68 
2022-12-01T10:19:39.7536993Z   Verifying  : kernel-headers-4.14.296-222.539.amzn2.x86_64               15/68 
2022-12-01T10:19:39.7635551Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-12-01T10:19:39.7724619Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            17/68 
2022-12-01T10:19:39.7807028Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           18/68 
2022-12-01T10:19:39.7899256Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             19/68 
2022-12-01T10:19:39.7997750Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              20/68 
2022-12-01T10:19:39.8083874Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            21/68 
2022-12-01T10:19:39.8172200Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               22/68 
2022-12-01T10:19:39.8264419Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     23/68 
2022-12-01T10:19:39.8370453Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       24/68 
2022-12-01T10:19:39.8460995Z   Verifying  : glibc-headers-2.26-62.amzn2.x86_64                         25/68 
2022-12-01T10:19:39.8538005Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      26/68 
2022-12-01T10:19:39.8634542Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              27/68 
2022-12-01T10:19:39.8718002Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          28/68 
2022-12-01T10:19:39.8800243Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             29/68 
2022-12-01T10:19:39.8887536Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          30/68 
2022-12-01T10:19:39.8996215Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  31/68 
2022-12-01T10:19:39.9090515Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       32/68 
2022-12-01T10:19:39.9172801Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  33/68 
2022-12-01T10:19:39.9249373Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  34/68 
2022-12-01T10:19:39.9332444Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      35/68 
2022-12-01T10:19:39.9422692Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           36/68 
2022-12-01T10:19:39.9528116Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 37/68 
2022-12-01T10:19:39.9620606Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           38/68 
2022-12-01T10:19:39.9719485Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      39/68 
2022-12-01T10:19:39.9804615Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               40/68 
2022-12-01T10:19:39.9888110Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          41/68 
2022-12-01T10:19:39.9976905Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             42/68 
2022-12-01T10:19:40.0063909Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          43/68 
2022-12-01T10:19:40.0172957Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              44/68 
2022-12-01T10:19:40.0258882Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        45/68 
2022-12-01T10:19:40.0356554Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               46/68 
2022-12-01T10:19:40.0451351Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        47/68 
2022-12-01T10:19:40.0553489Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  48/68 
2022-12-01T10:19:40.0631415Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          49/68 
2022-12-01T10:19:40.0713590Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-12-01T10:19:40.0802947Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   51/68 
2022-12-01T10:19:40.0896066Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              52/68 
2022-12-01T10:19:40.0982344Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        53/68 
2022-12-01T10:19:40.1060882Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              54/68 
2022-12-01T10:19:40.1142987Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         55/68 
2022-12-01T10:19:40.1246609Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             56/68 
2022-12-01T10:19:40.1333706Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           57/68 
2022-12-01T10:19:40.1438629Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                58/68 
2022-12-01T10:19:40.1525578Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               59/68 
2022-12-01T10:19:40.1612427Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    60/68 
2022-12-01T10:19:40.1714724Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          61/68 
2022-12-01T10:19:40.1802029Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               62/68 
2022-12-01T10:19:40.1892418Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 63/68 
2022-12-01T10:19:40.1995548Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         64/68 
2022-12-01T10:19:40.2089184Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              65/68 
2022-12-01T10:19:40.2173077Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              66/68 
2022-12-01T10:19:40.2253607Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   67/68 
2022-12-01T10:19:40.2993246Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         68/68 
2022-12-01T10:19:40.2993496Z 
2022-12-01T10:19:40.2993606Z Installed:
2022-12-01T10:19:40.2994171Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-12-01T10:19:40.2997382Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-12-01T10:19:40.2997851Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-12-01T10:19:40.2998283Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-12-01T10:19:40.2998709Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-12-01T10:19:40.2999115Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-12-01T10:19:40.3002652Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-12-01T10:19:40.3003106Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-12-01T10:19:40.3003571Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-12-01T10:19:40.3003999Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-12-01T10:19:40.3004417Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2022-12-01T10:19:40.3004839Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2022-12-01T10:19:40.3005268Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2022-12-01T10:19:40.3005701Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-12-01T10:19:40.3006096Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-12-01T10:19:40.3006504Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-12-01T10:19:40.3006898Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-12-01T10:19:40.3007334Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-12-01T10:19:40.3007751Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-12-01T10:19:40.3008148Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-12-01T10:19:40.3008587Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-12-01T10:19:40.3009002Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-12-01T10:19:40.3009450Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-12-01T10:19:40.3009909Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2022-12-01T10:19:40.3010353Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2022-12-01T10:19:40.3010533Z 
2022-12-01T10:19:40.3010646Z Dependency Installed:
2022-12-01T10:19:40.3011021Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-12-01T10:19:40.3011442Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-12-01T10:19:40.3011859Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-12-01T10:19:40.3012301Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-12-01T10:19:40.3012722Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2022-12-01T10:19:40.3013132Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-12-01T10:19:40.3013587Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-12-01T10:19:40.3014037Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-12-01T10:19:40.3014756Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2022-12-01T10:19:40.3015214Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-12-01T10:19:40.3015627Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-12-01T10:19:40.3016082Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-12-01T10:19:40.3016815Z   glibc-devel.x86_64 0:2.26-62.amzn2                                            
2022-12-01T10:19:40.3017313Z   glibc-headers.x86_64 0:2.26-62.amzn2                                          
2022-12-01T10:19:40.3017670Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-12-01T10:19:40.3018111Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2                                   
2022-12-01T10:19:40.3018545Z   kernel-devel.x86_64 0:4.14.296-222.539.amzn2                                  
2022-12-01T10:19:40.3018984Z   kernel-headers.x86_64 0:4.14.296-222.539.amzn2                                
2022-12-01T10:19:40.3019411Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2022-12-01T10:19:40.3019828Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2022-12-01T10:19:40.3020233Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2022-12-01T10:19:40.3020655Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2022-12-01T10:19:40.3021083Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-12-01T10:19:40.3021486Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-12-01T10:19:40.3021900Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2022-12-01T10:19:40.3022315Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-12-01T10:19:40.3022775Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2022-12-01T10:19:40.3023148Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2022-12-01T10:19:40.3023562Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-12-01T10:19:40.3023971Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-12-01T10:19:40.3024362Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-12-01T10:19:40.3024776Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-12-01T10:19:40.3025203Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-12-01T10:19:40.3025632Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-12-01T10:19:40.3026076Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-12-01T10:19:40.3026537Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-12-01T10:19:40.3027048Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-12-01T10:19:40.3027495Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-12-01T10:19:40.3027905Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-12-01T10:19:40.3028351Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2022-12-01T10:19:40.3028810Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2022-12-01T10:19:40.3029225Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-12-01T10:19:40.3029644Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2                                        
2022-12-01T10:19:40.3029845Z 
2022-12-01T10:19:40.3029950Z Complete!
2022-12-01T10:19:40.3378266Z ++ uname -r
2022-12-01T10:19:40.3384354Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-12-01T10:19:40.8482347Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:19:40.8593595Z Existing lock /var/run/yum.pid: another copy is running as pid 64208.
2022-12-01T10:19:40.8594034Z Another app is currently holding the yum lock; waiting for it to exit...
2022-12-01T10:19:40.8602454Z   The other application is: yum
2022-12-01T10:19:40.8603231Z     Memory :  91 M RSS (308 MB VSZ)
2022-12-01T10:19:40.8604093Z     Started: Thu Dec  1 10:19:38 2022 - 00:02 ago
2022-12-01T10:19:40.8604790Z     State  : Running, pid: 64208
2022-12-01T10:19:42.8629417Z Another app is currently holding the yum lock; waiting for it to exit...
2022-12-01T10:19:42.8635889Z   The other application is: yum
2022-12-01T10:19:42.8636175Z     Memory : 167 M RSS (386 MB VSZ)
2022-12-01T10:19:42.8637287Z     Started: Thu Dec  1 10:19:38 2022 - 00:04 ago
2022-12-01T10:19:42.8637592Z     State  : Running, pid: 64208
2022-12-01T10:19:45.1379791Z Resolving Dependencies
2022-12-01T10:19:45.1385481Z --> Running transaction check
2022-12-01T10:19:45.1385978Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-12-01T10:19:45.4256718Z --> Finished Dependency Resolution
2022-12-01T10:19:45.5074205Z 
2022-12-01T10:19:45.5074856Z Dependencies Resolved
2022-12-01T10:19:45.5080451Z 
2022-12-01T10:19:45.5080701Z ================================================================================
2022-12-01T10:19:45.5081082Z  Package          Arch       Version                       Repository      Size
2022-12-01T10:19:45.5081477Z ================================================================================
2022-12-01T10:19:45.5081759Z Installing:
2022-12-01T10:19:45.5082272Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-12-01T10:19:45.5082494Z 
2022-12-01T10:19:45.5082613Z Transaction Summary
2022-12-01T10:19:45.5082903Z ================================================================================
2022-12-01T10:19:45.5083157Z Install  1 Package
2022-12-01T10:19:45.5083318Z 
2022-12-01T10:19:45.5083446Z Total download size: 13 M
2022-12-01T10:19:45.5083712Z Installed size: 60 M
2022-12-01T10:19:45.5084102Z Downloading packages:
2022-12-01T10:19:45.5093039Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-12-01T10:19:45.9615092Z Running transaction check
2022-12-01T10:19:45.9806840Z Running transaction test
2022-12-01T10:19:46.3838939Z Transaction test succeeded
2022-12-01T10:19:46.3843235Z Running transaction
2022-12-01T10:20:01.7309607Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-12-01T10:20:01.8117643Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-12-01T10:20:01.8117932Z 
2022-12-01T10:20:01.8118042Z Installed:
2022-12-01T10:20:01.8118448Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-12-01T10:20:01.8118646Z 
2022-12-01T10:20:01.8118758Z Complete!
2022-12-01T10:20:01.8438278Z + sudo modprobe backlight
2022-12-01T10:20:01.8669397Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.57.run
2022-12-01T10:20:07.3335028Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-12-01T10:20:08.7020799Z Verifying archive integrity... OK
2022-12-01T10:20:35.3908373Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.57................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-12-01T10:20:35.5327982Z 
2022-12-01T10:20:35.5328915Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-12-01T10:20:35.5329690Z 
2022-12-01T10:20:51.5190992Z 
2022-12-01T10:20:51.5192926Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-12-01T10:20:51.5193672Z 
2022-12-01T10:20:58.7873594Z + sudo rm -fv /tmp/nvidia_driver
2022-12-01T10:20:58.8474254Z removed ‘/tmp/nvidia_driver’
2022-12-01T10:20:58.8489382Z + nvidia-smi
2022-12-01T10:21:03.4471545Z Thu Dec  1 10:21:03 2022       
2022-12-01T10:21:03.4472130Z +-----------------------------------------------------------------------------+
2022-12-01T10:21:03.4475086Z | NVIDIA-SMI 515.57       Driver Version: 515.57       CUDA Version: 11.7     |
2022-12-01T10:21:03.4475651Z |-------------------------------+----------------------+----------------------+
2022-12-01T10:21:03.4476185Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-12-01T10:21:03.4476691Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-12-01T10:21:03.4477059Z |                               |                      |               MIG M. |
2022-12-01T10:21:03.4477362Z |===============================+======================+======================|
2022-12-01T10:21:03.4518483Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |            585654386 |
2022-12-01T10:21:03.4518833Z | N/A   28C    P0    39W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-12-01T10:21:03.4519154Z |                               |                      |                  N/A |
2022-12-01T10:21:03.4519609Z +-------------------------------+----------------------+----------------------+
2022-12-01T10:21:03.4565283Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |           2021836848 |
2022-12-01T10:21:03.4565654Z | N/A   32C    P0    36W / 150W |      0MiB /  7680MiB |    100%      Default |
2022-12-01T10:21:03.4565974Z |                               |                      |                  N/A |
2022-12-01T10:21:03.4566417Z +-------------------------------+----------------------+----------------------+
2022-12-01T10:21:03.4566776Z                                                                                
2022-12-01T10:21:03.4567209Z +-----------------------------------------------------------------------------+
2022-12-01T10:21:03.4567586Z | Processes:                                                                  |
2022-12-01T10:21:03.4567912Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-12-01T10:21:03.4568262Z |        ID   ID                                                   Usage      |
2022-12-01T10:21:03.4568577Z |=============================================================================|
2022-12-01T10:21:03.4571037Z |  No running processes found                                                 |
2022-12-01T10:21:03.4571865Z +-----------------------------------------------------------------------------+
2022-12-01T10:21:03.9996698Z == Installing nvidia container toolkit for amzn2 ==
2022-12-01T10:21:04.0000319Z + sudo yum install -y yum-utils
2022-12-01T10:21:04.5493422Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:21:04.8283671Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-12-01T10:21:04.8284058Z Nothing to do
2022-12-01T10:21:04.8484997Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-12-01T10:21:05.3770989Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:21:05.4039609Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-12-01T10:21:05.4040525Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-12-01T10:21:05.4041195Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-12-01T10:21:05.4186169Z + sudo yum install -y nvidia-docker2
2022-12-01T10:21:05.9462077Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:21:05.9849863Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-12-01T10:21:05.9930090Z Importing GPG key 0xF796ECB0:
2022-12-01T10:21:05.9930493Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-12-01T10:21:05.9930900Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-12-01T10:21:05.9931359Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-12-01T10:21:06.3817706Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-12-01T10:21:06.3897669Z Importing GPG key 0xF796ECB0:
2022-12-01T10:21:06.3898069Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-12-01T10:21:06.3898498Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-12-01T10:21:06.3898987Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-12-01T10:21:06.6113197Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-12-01T10:21:06.6190866Z Importing GPG key 0xF796ECB0:
2022-12-01T10:21:06.6191292Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-12-01T10:21:06.6191671Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-12-01T10:21:06.6192135Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-12-01T10:21:08.3021645Z Resolving Dependencies
2022-12-01T10:21:08.3028166Z --> Running transaction check
2022-12-01T10:21:08.3028684Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2022-12-01T10:21:08.3054579Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2022-12-01T10:21:08.3435536Z --> Running transaction check
2022-12-01T10:21:08.3436066Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed
2022-12-01T10:21:08.3594980Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-12-01T10:21:08.3605403Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-12-01T10:21:08.3733156Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-12-01T10:21:08.3733645Z --> Running transaction check
2022-12-01T10:21:08.3734604Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed
2022-12-01T10:21:08.3745060Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-12-01T10:21:08.3771592Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-12-01T10:21:08.3772344Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-12-01T10:21:08.3772972Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed
2022-12-01T10:21:08.3775734Z --> Running transaction check
2022-12-01T10:21:08.3776179Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed
2022-12-01T10:21:08.6739835Z --> Finished Dependency Resolution
2022-12-01T10:21:08.7511705Z 
2022-12-01T10:21:08.7512245Z Dependencies Resolved
2022-12-01T10:21:08.7524723Z 
2022-12-01T10:21:08.7525228Z ================================================================================
2022-12-01T10:21:08.7525619Z  Package                        Arch    Version      Repository            Size
2022-12-01T10:21:08.7525964Z ================================================================================
2022-12-01T10:21:08.7526254Z Installing:
2022-12-01T10:21:08.7527002Z  nvidia-docker2                 noarch  2.11.0-1     libnvidia-container  8.7 k
2022-12-01T10:21:08.7527468Z Installing for dependencies:
2022-12-01T10:21:08.7527942Z  libnvidia-container-tools      x86_64  1.11.0-1     libnvidia-container   49 k
2022-12-01T10:21:08.7528460Z  libnvidia-container1           x86_64  1.11.0-1     libnvidia-container  1.0 M
2022-12-01T10:21:08.7528939Z  nvidia-container-toolkit       x86_64  1.11.0-1     libnvidia-container  780 k
2022-12-01T10:21:08.7529479Z  nvidia-container-toolkit-base  x86_64  1.11.0-1     libnvidia-container  2.5 M
2022-12-01T10:21:08.7529741Z 
2022-12-01T10:21:08.7529860Z Transaction Summary
2022-12-01T10:21:08.7530166Z ================================================================================
2022-12-01T10:21:08.7530455Z Install  1 Package (+4 Dependent packages)
2022-12-01T10:21:08.7530658Z 
2022-12-01T10:21:08.7530774Z Total download size: 4.3 M
2022-12-01T10:21:08.7531039Z Installed size: 12 M
2022-12-01T10:21:08.7531268Z Downloading packages:
2022-12-01T10:21:08.8652514Z --------------------------------------------------------------------------------
2022-12-01T10:21:08.8653219Z Total                                               38 MB/s | 4.3 MB  00:00     
2022-12-01T10:21:08.8699651Z Running transaction check
2022-12-01T10:21:08.8874525Z Running transaction test
2022-12-01T10:21:08.9040539Z Transaction test succeeded
2022-12-01T10:21:08.9043947Z Running transaction
2022-12-01T10:21:09.4142013Z   Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64                1/5 
2022-12-01T10:21:09.4492379Z   Installing : libnvidia-container1-1.11.0-1.x86_64                         2/5 
2022-12-01T10:21:09.5574993Z   Installing : libnvidia-container-tools-1.11.0-1.x86_64                    3/5 
2022-12-01T10:21:09.5784511Z   Installing : nvidia-container-toolkit-1.11.0-1.x86_64                     4/5 
2022-12-01T10:21:09.6129423Z   Installing : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-12-01T10:21:09.6233065Z   Verifying  : libnvidia-container1-1.11.0-1.x86_64                         1/5 
2022-12-01T10:21:09.6327824Z   Verifying  : nvidia-container-toolkit-base-1.11.0-1.x86_64                2/5 
2022-12-01T10:21:09.6419067Z   Verifying  : nvidia-container-toolkit-1.11.0-1.x86_64                     3/5 
2022-12-01T10:21:09.6516495Z   Verifying  : libnvidia-container-tools-1.11.0-1.x86_64                    4/5 
2022-12-01T10:21:09.7272422Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-12-01T10:21:09.7272725Z 
2022-12-01T10:21:09.7272833Z Installed:
2022-12-01T10:21:09.7273232Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2022-12-01T10:21:09.7273447Z 
2022-12-01T10:21:09.7273574Z Dependency Installed:
2022-12-01T10:21:09.7273975Z   libnvidia-container-tools.x86_64 0:1.11.0-1                                   
2022-12-01T10:21:09.7274450Z   libnvidia-container1.x86_64 0:1.11.0-1                                        
2022-12-01T10:21:09.7274915Z   nvidia-container-toolkit.x86_64 0:1.11.0-1                                    
2022-12-01T10:21:09.7275418Z   nvidia-container-toolkit-base.x86_64 0:1.11.0-1                               
2022-12-01T10:21:09.7275655Z 
2022-12-01T10:21:09.7275764Z Complete!
2022-12-01T10:21:09.8277371Z + sudo systemctl restart docker
2022-12-01T10:21:17.5889843Z + echo 'GPU_FLAG=--gpus all'
2022-12-01T10:21:18.3727535Z Command completed after 1 attempt(s).
2022-12-01T10:21:18.3727870Z 
2022-12-01T10:21:18.3730478Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:21:18.3779856Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-12-01T10:21:18.3780254Z python3 -m pip install psutil==5.9.1
2022-12-01T10:21:18.3780565Z python3 -m pip install pynvml==11.4.1
2022-12-01T10:21:18.3780920Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-12-01T10:21:18.3781440Z echo "::set-output name=monitor-script-pid::${!}"
2022-12-01T10:21:18.3795230Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:21:18.3795527Z env:
2022-12-01T10:21:18.3795770Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:21:18.3796023Z   GPU_FLAG: --gpus all
2022-12-01T10:21:18.3796274Z ##[endgroup]
2022-12-01T10:21:19.1374151Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:21:19.5116194Z Collecting psutil==5.9.1
2022-12-01T10:21:19.5311276Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2022-12-01T10:21:19.5988824Z Installing collected packages: psutil
2022-12-01T10:21:19.7544879Z Successfully installed psutil-5.9.1
2022-12-01T10:21:20.2264776Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:21:20.2913396Z Collecting pynvml==11.4.1
2022-12-01T10:21:20.3053520Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2022-12-01T10:21:20.3531884Z Installing collected packages: pynvml
2022-12-01T10:21:20.4066119Z Successfully installed pynvml-11.4.1
2022-12-01T10:21:20.4524339Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:21:20.4577732Z Prepare all required actions
2022-12-01T10:21:20.4578150Z Getting action download info
2022-12-01T10:21:20.6515615Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-12-01T10:21:20.8364152Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-12-01T10:21:20.9828182Z ##[group]Run ./.github/actions/download-build-artifacts
2022-12-01T10:21:20.9828463Z with:
2022-12-01T10:21:20.9828720Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:21:20.9828991Z env:
2022-12-01T10:21:20.9829208Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:21:20.9829467Z   GPU_FLAG: --gpus all
2022-12-01T10:21:20.9829694Z ##[endgroup]
2022-12-01T10:21:21.0149904Z ##[group]Run seemethere/download-artifact-s3@v4
2022-12-01T10:21:21.0150198Z with:
2022-12-01T10:21:21.0150457Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:21:21.0150762Z   s3-bucket: gha-artifacts
2022-12-01T10:21:21.0151059Z   region: us-east-1
2022-12-01T10:21:21.0151295Z env:
2022-12-01T10:21:21.0151527Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:21:21.0151771Z   GPU_FLAG: --gpus all
2022-12-01T10:21:21.0152018Z ##[endgroup]
2022-12-01T10:21:21.5577111Z Found 1 objects with prefix pytorch/pytorch/3591403534/linux-bionic-cuda11.6-py3.10-gcc7/
2022-12-01T10:21:21.5577747Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-12-01T10:21:27.7963807Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-12-01T10:21:27.7964174Z 
2022-12-01T10:21:27.7965995Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:21:27.7967125Z Artifact download has finished successfully
2022-12-01T10:21:27.8137421Z ##[group]Run unzip -o artifacts.zip
2022-12-01T10:21:27.8137754Z unzip -o artifacts.zip
2022-12-01T10:21:27.8151410Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:21:27.8151715Z env:
2022-12-01T10:21:27.8151956Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:21:27.8152209Z   GPU_FLAG: --gpus all
2022-12-01T10:21:27.8152459Z ##[endgroup]
2022-12-01T10:21:27.8230786Z Archive:  artifacts.zip
2022-12-01T10:21:27.8232687Z    creating: dist/
2022-12-01T10:21:29.8877524Z   inflating: dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl  
2022-12-01T10:21:29.8878317Z    creating: build/custom_test_artifacts/
2022-12-01T10:21:29.8878786Z    creating: build/custom_test_artifacts/custom-op-build/
2022-12-01T10:21:29.8879293Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-12-01T10:21:29.8885539Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:21:29.8886102Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-12-01T10:21:29.8886685Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:21:29.8887266Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:21:29.8887815Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:21:29.8890088Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:21:29.8891180Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:21:29.8891776Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:21:29.8892659Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:21:29.8896113Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:21:29.8897392Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:21:29.8898822Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:21:29.8899891Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:21:29.8901388Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:21:29.8903496Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:21:29.8904311Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-12-01T10:21:29.8904899Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-12-01T10:21:29.8959006Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-12-01T10:21:29.8959753Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-12-01T10:21:29.8960499Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-12-01T10:21:29.8961261Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-12-01T10:21:29.8961988Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-12-01T10:21:29.8962704Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-12-01T10:21:29.8963418Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-12-01T10:21:29.8964127Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-12-01T10:21:29.8965075Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-12-01T10:21:29.9007083Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-12-01T10:21:29.9048320Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-12-01T10:21:29.9049413Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-12-01T10:21:29.9050262Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-12-01T10:21:29.9050940Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-12-01T10:21:29.9051595Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-12-01T10:21:29.9052593Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-12-01T10:21:29.9053265Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-12-01T10:21:29.9055620Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-12-01T10:21:29.9128291Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-12-01T10:21:29.9201265Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-12-01T10:21:29.9201925Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-12-01T10:21:29.9202758Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-12-01T10:21:29.9203917Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-12-01T10:21:29.9204539Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:21:29.9205070Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-12-01T10:21:29.9205648Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-12-01T10:21:29.9206276Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-12-01T10:21:29.9206893Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-12-01T10:21:29.9207482Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-12-01T10:21:29.9208063Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-12-01T10:21:29.9209257Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-12-01T10:21:29.9209878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-12-01T10:21:29.9210500Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-12-01T10:21:29.9211060Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-12-01T10:21:29.9233096Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-12-01T10:21:29.9346551Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-12-01T10:21:29.9347134Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-12-01T10:21:29.9347728Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-12-01T10:21:29.9348395Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-12-01T10:21:29.9349019Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-12-01T10:21:29.9349633Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-12-01T10:21:29.9350230Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-12-01T10:21:29.9351031Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-12-01T10:21:29.9351809Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-12-01T10:21:29.9352435Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-12-01T10:21:29.9353561Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-12-01T10:21:29.9374594Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-12-01T10:21:29.9456477Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-12-01T10:21:29.9457141Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:21:29.9457758Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:21:29.9458310Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-12-01T10:21:29.9459049Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-12-01T10:21:29.9460194Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:21:29.9460741Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-12-01T10:21:29.9463912Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-12-01T10:21:29.9464590Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-12-01T10:21:29.9465307Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-12-01T10:21:29.9557700Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-12-01T10:21:29.9619960Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-12-01T10:21:29.9620745Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-12-01T10:21:29.9621207Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-12-01T10:21:29.9627635Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:21:29.9628182Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-12-01T10:21:29.9628737Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:21:29.9629290Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:21:29.9629850Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:21:29.9631951Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:21:29.9633050Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:21:29.9633840Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:21:29.9634419Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:21:29.9636680Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:21:29.9637748Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:21:29.9639646Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:21:29.9640540Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:21:29.9641653Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:21:29.9642986Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:21:29.9643869Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-12-01T10:21:29.9644587Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-12-01T10:21:29.9698418Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-12-01T10:21:29.9699614Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-12-01T10:21:29.9700561Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-12-01T10:21:29.9701730Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-12-01T10:21:29.9702689Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-12-01T10:21:29.9703611Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-12-01T10:21:29.9704524Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-12-01T10:21:29.9705246Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-12-01T10:21:29.9706077Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-12-01T10:21:29.9746471Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-12-01T10:21:29.9787642Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-12-01T10:21:29.9788729Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-12-01T10:21:29.9789596Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-12-01T10:21:29.9790275Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-12-01T10:21:29.9790912Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-12-01T10:21:29.9791779Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-12-01T10:21:29.9792824Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-12-01T10:21:29.9794852Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-12-01T10:21:29.9867732Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-12-01T10:21:29.9940809Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-12-01T10:21:29.9941469Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-12-01T10:21:29.9942015Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-12-01T10:21:29.9942565Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-12-01T10:21:29.9943559Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:21:29.9944376Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-12-01T10:21:29.9945270Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-12-01T10:21:29.9946221Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-12-01T10:21:29.9947064Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-12-01T10:21:29.9947807Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-12-01T10:21:29.9948731Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-12-01T10:21:29.9949892Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-12-01T10:21:29.9950638Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-12-01T10:21:29.9951229Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-12-01T10:21:29.9951831Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-12-01T10:21:29.9971029Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-12-01T10:21:30.0034493Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-12-01T10:21:30.0035552Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:21:30.0036324Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:21:30.0037108Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-12-01T10:21:30.0037783Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-12-01T10:21:30.0038685Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:21:30.0039828Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-12-01T10:21:30.0041479Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-12-01T10:21:30.0042528Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-12-01T10:21:30.0043640Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-12-01T10:21:30.0092397Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-12-01T10:21:30.0093456Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-12-01T10:21:30.0093997Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-12-01T10:21:30.0100364Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:21:30.0101680Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-12-01T10:21:30.0102405Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:21:30.0102982Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:21:30.0103755Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:21:30.0105189Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:21:30.0106596Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:21:30.0107335Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:21:30.0107934Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:21:30.0109363Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:21:30.0111045Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:21:30.0112720Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:21:30.0113910Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:21:30.0115386Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:21:30.0116780Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:21:30.0117388Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-12-01T10:21:30.0117990Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-12-01T10:21:30.0171031Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-12-01T10:21:30.0172709Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-12-01T10:21:30.0174621Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-12-01T10:21:30.0176393Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-12-01T10:21:30.0177642Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-12-01T10:21:30.0178380Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-12-01T10:21:30.0179230Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-12-01T10:21:30.0179962Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-12-01T10:21:30.0180713Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-12-01T10:21:30.0219327Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-12-01T10:21:30.0260429Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-12-01T10:21:30.0262024Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-12-01T10:21:30.0263574Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-12-01T10:21:30.0265075Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-12-01T10:21:30.0266173Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-12-01T10:21:30.0267599Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-12-01T10:21:30.0268303Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-12-01T10:21:30.0268986Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-12-01T10:21:30.0340274Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-12-01T10:21:30.0412971Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-12-01T10:21:30.0414739Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-12-01T10:21:30.0416043Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-12-01T10:21:30.0417272Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-12-01T10:21:30.0418494Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:21:30.0419789Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-12-01T10:21:30.0421327Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-12-01T10:21:30.0422860Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-12-01T10:21:30.0423697Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-12-01T10:21:30.0424331Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-12-01T10:21:30.0424968Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-12-01T10:21:30.0425605Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-12-01T10:21:30.0426226Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-12-01T10:21:30.0427308Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-12-01T10:21:30.0428468Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-12-01T10:21:30.0429153Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-12-01T10:21:30.0575130Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-12-01T10:21:30.0576568Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-12-01T10:21:30.0577994Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-12-01T10:21:30.0579480Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-12-01T10:21:30.0580911Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-12-01T10:21:30.0582052Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-12-01T10:21:30.0582725Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-12-01T10:21:30.0583389Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-12-01T10:21:30.0584031Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-12-01T10:21:30.0584685Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-12-01T10:21:30.0585334Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-12-01T10:21:30.0602440Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-12-01T10:21:30.0660833Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-12-01T10:21:30.0662366Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:21:30.0663735Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:21:30.0665091Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-12-01T10:21:30.0665992Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-12-01T10:21:30.0666561Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:21:30.0667149Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-12-01T10:21:30.0668280Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-12-01T10:21:30.0669614Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-12-01T10:21:30.0670485Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-12-01T10:21:30.0788609Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-12-01T10:21:30.0834197Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-12-01T10:21:30.0834955Z    creating: build/lib/
2022-12-01T10:21:30.0835594Z   inflating: build/lib/libclog.a     
2022-12-01T10:21:30.0901339Z   inflating: build/lib/libgtest.a    
2022-12-01T10:21:30.0911425Z   inflating: build/lib/libpthreadpool.a  
2022-12-01T10:21:30.1017543Z   inflating: build/lib/libprotobuf-lite.a  
2022-12-01T10:21:30.1108726Z   inflating: build/lib/libbenchmark.a  
2022-12-01T10:21:30.1117893Z   inflating: build/lib/libittnotify.a  
2022-12-01T10:21:30.1194611Z   inflating: build/lib/libasmjit.a   
2022-12-01T10:21:30.1728218Z   inflating: build/lib/libprotobuf.a  
2022-12-01T10:21:30.1760569Z   inflating: build/lib/libtensorpipe_uv.a  
2022-12-01T10:21:30.1892966Z   inflating: build/lib/libgloo.a     
2022-12-01T10:21:30.1912614Z   inflating: build/lib/libfmt.a      
2022-12-01T10:21:30.1914216Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-12-01T10:21:30.1915117Z   inflating: build/lib/libfoxi_loader.a  
2022-12-01T10:21:30.1991986Z   inflating: build/lib/libc10.so     
2022-12-01T10:21:30.1993935Z   inflating: build/lib/libtorch_global_deps.so  
2022-12-01T10:21:30.2022995Z   inflating: build/lib/libcpuinfo.a  
2022-12-01T10:21:30.2593019Z   inflating: build/lib/libprotoc.a   
2022-12-01T10:21:30.2601932Z   inflating: build/lib/libcpuinfo_internals.a  
2022-12-01T10:21:30.2618043Z   inflating: build/lib/libqnnpack.a  
2022-12-01T10:21:30.2641924Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-12-01T10:21:30.2644723Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-12-01T10:21:30.2666955Z   inflating: build/lib/libnnpack.a   
2022-12-01T10:21:30.2685944Z   inflating: build/lib/libgmock.a    
2022-12-01T10:21:30.2686597Z   inflating: build/lib/libgtest_main.a  
2022-12-01T10:21:30.2687448Z   inflating: build/lib/libbenchmark_main.a  
2022-12-01T10:21:31.0803145Z   inflating: build/lib/libdnnl.a     
2022-12-01T10:21:31.0944609Z   inflating: build/lib/libXNNPACK.a  
2022-12-01T10:21:31.1599725Z   inflating: build/lib/libtensorpipe.a  
2022-12-01T10:21:31.1641683Z   inflating: build/lib/libc10_cuda.so  
2022-12-01T10:21:31.3183675Z   inflating: build/lib/libfbgemm.a   
2022-12-01T10:21:31.3184328Z   inflating: build/lib/libgmock_main.a  
2022-12-01T10:21:31.4313816Z   inflating: build/lib/libdnnl_graph.a  
2022-12-01T10:21:31.4735652Z   inflating: build/lib/libkineto.a   
2022-12-01T10:21:31.5025128Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-12-01T10:21:31.5070297Z   inflating: build/lib/libcaffe2_protos.a  
2022-12-01T10:21:31.5118205Z   inflating: build/lib/libonnx_proto.a  
2022-12-01T10:21:31.5796539Z   inflating: build/lib/libonnx.a     
2022-12-01T10:21:31.6225767Z   inflating: build/lib/libgloo_cuda.a  
2022-12-01T10:21:33.9426068Z   inflating: build/lib/libtorch_cpu.so  
2022-12-01T10:21:34.2863275Z   inflating: build/lib/libtorch_cuda_cpp.so  
2022-12-01T10:21:36.0534863Z   inflating: build/lib/libtorch_cuda_cu.so  
2022-12-01T10:21:36.0535851Z   inflating: build/lib/libtorch_cuda.so  
2022-12-01T10:21:36.0536838Z   inflating: build/lib/libtorch.so   
2022-12-01T10:21:37.0425632Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-12-01T10:21:37.0427921Z   inflating: build/lib/libc10d_cuda_test.so  
2022-12-01T10:21:37.0451698Z   inflating: build/lib/libjitbackend_test.so  
2022-12-01T10:21:37.0512024Z   inflating: build/lib/libtorchbind_test.so  
2022-12-01T10:21:37.0542824Z   inflating: build/lib/libbackend_with_compiler.so  
2022-12-01T10:21:37.0547635Z   inflating: build/lib/libshm.so     
2022-12-01T10:21:37.2327892Z   inflating: build/lib/libtorch_python.so  
2022-12-01T10:21:37.2367255Z   inflating: build/lib/libnnapi_backend.so  
2022-12-01T10:21:37.2367936Z    creating: build/bin/
2022-12-01T10:21:37.2420138Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-12-01T10:21:37.2475266Z   inflating: build/bin/c10_DeviceGuard_test  
2022-12-01T10:21:37.2529406Z   inflating: build/bin/c10_Device_test  
2022-12-01T10:21:37.2592064Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-12-01T10:21:37.2643251Z   inflating: build/bin/c10_StreamGuard_test  
2022-12-01T10:21:37.2696789Z   inflating: build/bin/c10_SymInt_test  
2022-12-01T10:21:37.2756151Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-12-01T10:21:37.2815803Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-12-01T10:21:37.2876666Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-12-01T10:21:37.2927803Z   inflating: build/bin/c10_Array_test  
2022-12-01T10:21:37.2984558Z   inflating: build/bin/c10_Bitset_test  
2022-12-01T10:21:37.3036014Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-12-01T10:21:37.3090730Z   inflating: build/bin/c10_C++17_test  
2022-12-01T10:21:37.3143400Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-12-01T10:21:37.3195996Z   inflating: build/bin/c10_Half_test  
2022-12-01T10:21:37.3256642Z   inflating: build/bin/c10_LeftRight_test  
2022-12-01T10:21:37.3323298Z   inflating: build/bin/c10_Metaprogramming_test  
2022-12-01T10:21:37.3479122Z   inflating: build/bin/c10_SmallVectorTest  
2022-12-01T10:21:37.3532921Z   inflating: build/bin/c10_Synchronized_test  
2022-12-01T10:21:37.3593812Z   inflating: build/bin/c10_ThreadLocal_test  
2022-12-01T10:21:37.3649622Z   inflating: build/bin/c10_TypeIndex_test  
2022-12-01T10:21:37.3701162Z   inflating: build/bin/c10_TypeTraits_test  
2022-12-01T10:21:37.3756308Z   inflating: build/bin/c10_accumulate_test  
2022-12-01T10:21:37.3810008Z   inflating: build/bin/c10_TypeList_test  
2022-12-01T10:21:37.3869695Z   inflating: build/bin/c10_bfloat16_test  
2022-12-01T10:21:37.3927091Z   inflating: build/bin/c10_complex_math_test  
2022-12-01T10:21:37.3986506Z   inflating: build/bin/c10_complex_test  
2022-12-01T10:21:37.4104609Z   inflating: build/bin/c10_either_test  
2022-12-01T10:21:37.4160262Z   inflating: build/bin/c10_exception_test  
2022-12-01T10:21:37.4213550Z   inflating: build/bin/c10_flags_test  
2022-12-01T10:21:37.4267607Z   inflating: build/bin/c10_irange_test  
2022-12-01T10:21:37.4450201Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-12-01T10:21:37.4511941Z   inflating: build/bin/c10_logging_test  
2022-12-01T10:21:37.4591872Z   inflating: build/bin/c10_optional_test  
2022-12-01T10:21:37.4658382Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-12-01T10:21:37.4717121Z   inflating: build/bin/c10_registry_test  
2022-12-01T10:21:37.4780762Z   inflating: build/bin/c10_string_view_test  
2022-12-01T10:21:37.4835509Z   inflating: build/bin/c10_tempfile_test  
2022-12-01T10:21:37.4896046Z   inflating: build/bin/c10_typeid_test  
2022-12-01T10:21:37.4956275Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-12-01T10:21:37.5478044Z   inflating: build/bin/protoc-3.13.0.0  
2022-12-01T10:21:37.5999509Z   inflating: build/bin/protoc        
2022-12-01T10:21:37.6051349Z   inflating: build/bin/c10_cuda_CUDATest  
2022-12-01T10:21:37.6369087Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-12-01T10:21:37.6723526Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-12-01T10:21:37.6780762Z   inflating: build/bin/HashStoreTest  
2022-12-01T10:21:37.6845363Z   inflating: build/bin/TCPStoreTest  
2022-12-01T10:21:37.6902637Z   inflating: build/bin/FileStoreTest  
2022-12-01T10:21:37.6918506Z   inflating: build/bin/ProcessGroupMPITest  
2022-12-01T10:21:37.6921672Z   inflating: build/bin/example_allreduce  
2022-12-01T10:21:37.6977691Z   inflating: build/bin/Dimname_test  
2022-12-01T10:21:37.7056225Z   inflating: build/bin/Dict_test     
2022-12-01T10:21:37.7124106Z   inflating: build/bin/MaybeOwned_test  
2022-12-01T10:21:37.7187454Z   inflating: build/bin/apply_utils_test  
2022-12-01T10:21:37.7248306Z   inflating: build/bin/NamedTensor_test  
2022-12-01T10:21:37.7311662Z   inflating: build/bin/atest         
2022-12-01T10:21:37.7369547Z   inflating: build/bin/broadcast_test  
2022-12-01T10:21:37.7435134Z   inflating: build/bin/basic         
2022-12-01T10:21:37.7497720Z   inflating: build/bin/cpu_generator_test  
2022-12-01T10:21:37.7553622Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-12-01T10:21:37.7606702Z   inflating: build/bin/dispatch_key_set_test  
2022-12-01T10:21:37.7701712Z   inflating: build/bin/cpu_rng_test  
2022-12-01T10:21:37.7754718Z   inflating: build/bin/dlconvertor_test  
2022-12-01T10:21:37.7816985Z   inflating: build/bin/extension_backend_test  
2022-12-01T10:21:37.7876652Z   inflating: build/bin/half_test     
2022-12-01T10:21:37.7929136Z   inflating: build/bin/lazy_tensor_test  
2022-12-01T10:21:37.8031011Z   inflating: build/bin/ivalue_test   
2022-12-01T10:21:37.8088466Z   inflating: build/bin/math_kernel_test  
2022-12-01T10:21:37.8145710Z   inflating: build/bin/memory_format_test  
2022-12-01T10:21:37.8201915Z   inflating: build/bin/memory_overlapping_test  
2022-12-01T10:21:37.8256604Z   inflating: build/bin/operator_name_test  
2022-12-01T10:21:37.8316520Z   inflating: build/bin/native_test   
2022-12-01T10:21:37.8372097Z   inflating: build/bin/mobile_memory_cleanup  
2022-12-01T10:21:37.8425500Z   inflating: build/bin/operators_test  
2022-12-01T10:21:37.8481467Z   inflating: build/bin/packedtensoraccessor_test  
2022-12-01T10:21:37.8551934Z   inflating: build/bin/pow_test      
2022-12-01T10:21:37.8612925Z   inflating: build/bin/quantized_test  
2022-12-01T10:21:37.8667150Z   inflating: build/bin/reportMemoryUsage_test  
2022-12-01T10:21:37.8719567Z   inflating: build/bin/reduce_ops_test  
2022-12-01T10:21:37.8780237Z   inflating: build/bin/scalar_tensor_test  
2022-12-01T10:21:37.8843235Z   inflating: build/bin/scalar_test   
2022-12-01T10:21:37.8898894Z   inflating: build/bin/stride_properties_test  
2022-12-01T10:21:37.8901448Z   inflating: build/bin/thread_init_test  
2022-12-01T10:21:37.8986158Z   inflating: build/bin/tensor_iterator_test  
2022-12-01T10:21:37.9045382Z   inflating: build/bin/type_ptr_test  
2022-12-01T10:21:37.9105039Z   inflating: build/bin/test_parallel  
2022-12-01T10:21:37.9169713Z   inflating: build/bin/type_test     
2022-12-01T10:21:37.9222254Z   inflating: build/bin/variant_test  
2022-12-01T10:21:37.9278021Z   inflating: build/bin/undefined_tensor_test  
2022-12-01T10:21:37.9352471Z   inflating: build/bin/vmap_test     
2022-12-01T10:21:37.9353537Z   inflating: build/bin/verify_api_visibility  
2022-12-01T10:21:37.9408526Z   inflating: build/bin/weakref_test  
2022-12-01T10:21:37.9462682Z   inflating: build/bin/wrapdim_test  
2022-12-01T10:21:37.9526389Z   inflating: build/bin/IListRef_test  
2022-12-01T10:21:37.9578681Z   inflating: build/bin/xla_tensor_test  
2022-12-01T10:21:37.9696583Z   inflating: build/bin/List_test     
2022-12-01T10:21:37.9766372Z   inflating: build/bin/KernelFunction_test  
2022-12-01T10:21:37.9897529Z   inflating: build/bin/kernel_function_legacy_test  
2022-12-01T10:21:38.0001360Z   inflating: build/bin/kernel_function_test  
2022-12-01T10:21:38.0140990Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-12-01T10:21:38.0252424Z   inflating: build/bin/kernel_lambda_test  
2022-12-01T10:21:38.0316618Z   inflating: build/bin/kernel_stackbased_test  
2022-12-01T10:21:38.0419968Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-12-01T10:21:38.0473721Z   inflating: build/bin/CppSignature_test  
2022-12-01T10:21:38.0524946Z   inflating: build/bin/op_allowlist_test  
2022-12-01T10:21:38.0581748Z   inflating: build/bin/inline_container_test  
2022-12-01T10:21:38.0894756Z   inflating: build/bin/op_registration_test  
2022-12-01T10:21:38.0954954Z   inflating: build/bin/backend_fallback_test  
2022-12-01T10:21:38.1010547Z   inflating: build/bin/cuda_apply_test  
2022-12-01T10:21:38.1067720Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-12-01T10:21:38.1132361Z   inflating: build/bin/cuda_atomic_ops_test  
2022-12-01T10:21:38.1204976Z   inflating: build/bin/cuda_complex_math_test  
2022-12-01T10:21:38.1257298Z   inflating: build/bin/cuda_device_test  
2022-12-01T10:21:38.1319632Z   inflating: build/bin/cuda_complex_test  
2022-12-01T10:21:38.1383752Z   inflating: build/bin/cuda_cub_test  
2022-12-01T10:21:38.1437011Z   inflating: build/bin/cuda_dlconvertor_test  
2022-12-01T10:21:38.1491094Z   inflating: build/bin/cuda_integer_divider_test  
2022-12-01T10:21:38.1563280Z   inflating: build/bin/cuda_distributions_test  
2022-12-01T10:21:38.1625802Z   inflating: build/bin/cuda_generator_test  
2022-12-01T10:21:38.1678482Z   inflating: build/bin/cuda_half_test  
2022-12-01T10:21:38.1730319Z   inflating: build/bin/cuda_optional_test  
2022-12-01T10:21:38.1786432Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-12-01T10:21:38.1851169Z   inflating: build/bin/cuda_stream_test  
2022-12-01T10:21:38.1905937Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-12-01T10:21:38.1957623Z   inflating: build/bin/cuda_cudnn_test  
2022-12-01T10:21:38.2013285Z   inflating: build/bin/cuda_vectorized_test  
2022-12-01T10:21:38.2031279Z   inflating: build/bin/tutorial_tensorexpr  
2022-12-01T10:21:38.2101105Z   inflating: build/bin/ProcessGroupGlooTest  
2022-12-01T10:21:38.2163392Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-12-01T10:21:38.2229274Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-12-01T10:21:38.2291401Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-12-01T10:21:38.2348070Z   inflating: build/bin/ProcessGroupUCCTest  
2022-12-01T10:21:38.2405941Z   inflating: build/bin/test_dist_autograd  
2022-12-01T10:21:38.2408558Z   inflating: build/bin/parallel_benchmark  
2022-12-01T10:21:38.2483314Z   inflating: build/bin/test_cpp_rpc  
2022-12-01T10:21:38.2557424Z   inflating: build/bin/test_mobile_nnc  
2022-12-01T10:21:38.3477484Z   inflating: build/bin/test_tensorexpr  
2022-12-01T10:21:38.3488507Z   inflating: build/bin/aot_model_compiler_test  
2022-12-01T10:21:38.3872724Z   inflating: build/bin/test_lazy     
2022-12-01T10:21:38.3878026Z   inflating: build/bin/torch_shm_manager  
2022-12-01T10:21:38.5198773Z   inflating: build/bin/test_api      
2022-12-01T10:21:38.6368469Z   inflating: build/bin/test_jit      
2022-12-01T10:21:38.6370121Z   inflating: .pytorch-test-times.json  
2022-12-01T10:21:38.6401209Z ##[group]Run df -H
2022-12-01T10:21:38.6401469Z df -H
2022-12-01T10:21:38.6415433Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:21:38.6415740Z env:
2022-12-01T10:21:38.6415981Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:21:38.6416232Z   GPU_FLAG: --gpus all
2022-12-01T10:21:38.6416483Z ##[endgroup]
2022-12-01T10:21:38.6456474Z Filesystem      Size  Used Avail Use% Mounted on
2022-12-01T10:21:38.6456838Z devtmpfs        129G     0  129G   0% /dev
2022-12-01T10:21:38.6457133Z tmpfs           129G     0  129G   0% /dev/shm
2022-12-01T10:21:38.6457402Z tmpfs           129G  607k  129G   1% /run
2022-12-01T10:21:38.6457696Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-12-01T10:21:38.6458338Z /dev/xvda1      162G   30G  132G  19% /
2022-12-01T10:21:38.6458635Z tmpfs            26G     0   26G   0% /run/user/0
2022-12-01T10:21:38.6482024Z ##[group]Run .github/scripts/parse_ref.py
2022-12-01T10:21:38.6482361Z .github/scripts/parse_ref.py
2022-12-01T10:21:38.6493812Z shell: /usr/bin/bash -e {0}
2022-12-01T10:21:38.6494066Z env:
2022-12-01T10:21:38.6494284Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:21:38.6494778Z   GPU_FLAG: --gpus all
2022-12-01T10:21:38.6495030Z ##[endgroup]
2022-12-01T10:21:38.6740217Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:21:38.6802624Z ##[group]Run set -x
2022-12-01T10:21:38.6802996Z set -x
2022-12-01T10:21:38.6803219Z 
2022-12-01T10:21:38.6803607Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-12-01T10:21:38.6803956Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-12-01T10:21:38.6804304Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-12-01T10:21:38.6804610Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-12-01T10:21:38.6804884Z else
2022-12-01T10:21:38.6805161Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-12-01T10:21:38.6805414Z fi
2022-12-01T10:21:38.6805631Z 
2022-12-01T10:21:38.6805947Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-12-01T10:21:38.6806242Z 
2022-12-01T10:21:38.6806538Z # sanitize the input commit message and PR body here:
2022-12-01T10:21:38.6806827Z #
2022-12-01T10:21:38.6807185Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-12-01T10:21:38.6807689Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-12-01T10:21:38.6808112Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-12-01T10:21:38.6808419Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-12-01T10:21:38.6808655Z 
2022-12-01T10:21:38.6809002Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-12-01T10:21:38.6809373Z # wreak havoc internally
2022-12-01T10:21:38.6809674Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-12-01T10:21:38.6810000Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-12-01T10:21:38.6810257Z 
2022-12-01T10:21:38.6810561Z # detached container should get cleaned up by teardown_ec2_linux
2022-12-01T10:21:38.6810943Z # TODO: Stop building test binaries as part of the build phase
2022-12-01T10:21:38.6811314Z # Used for GPU_FLAG since that doesn't play nice
2022-12-01T10:21:38.6811638Z # shellcheck disable=SC2086,SC2090
2022-12-01T10:21:38.6811927Z container_name=$(docker run \
2022-12-01T10:21:38.6812201Z   ${GPU_FLAG:-} \
2022-12-01T10:21:38.6812467Z   -e BUILD_ENVIRONMENT \
2022-12-01T10:21:38.6812721Z   -e PR_NUMBER \
2022-12-01T10:21:38.6812984Z   -e GITHUB_ACTIONS \
2022-12-01T10:21:38.6813239Z   -e BASE_SHA \
2022-12-01T10:21:38.6813484Z   -e BRANCH \
2022-12-01T10:21:38.6813705Z   -e SHA1 \
2022-12-01T10:21:38.6813959Z   -e AWS_DEFAULT_REGION \
2022-12-01T10:21:38.6814225Z   -e IN_WHEEL_TEST \
2022-12-01T10:21:38.6814730Z   -e SHARD_NUMBER \
2022-12-01T10:21:38.6815004Z   -e TEST_CONFIG \
2022-12-01T10:21:38.6815261Z   -e NUM_TEST_SHARDS \
2022-12-01T10:21:38.6815500Z   -e PR_BODY \
2022-12-01T10:21:38.6815761Z   -e COMMIT_MESSAGES \
2022-12-01T10:21:38.6816049Z   -e PYTORCH_RETRY_TEST_CASES \
2022-12-01T10:21:38.6816342Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-12-01T10:21:38.6816631Z   -e PR_LABELS \
2022-12-01T10:21:38.6816914Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-12-01T10:21:38.6817183Z   -e SCCACHE_BUCKET \
2022-12-01T10:21:38.6817457Z   -e SCCACHE_S3_KEY_PREFIX \
2022-12-01T10:21:38.6817721Z   -e XLA_CUDA \
2022-12-01T10:21:38.6817999Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-12-01T10:21:38.6818323Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-12-01T10:21:38.6818644Z   --ulimit stack=10485760:83886080 \
2022-12-01T10:21:38.6818950Z   --security-opt seccomp=unconfined \
2022-12-01T10:21:38.6819236Z   --cap-add=SYS_PTRACE \
2022-12-01T10:21:38.6819500Z   --ipc=host \
2022-12-01T10:21:38.6819868Z   --shm-size="${SHM_SIZE}" \
2022-12-01T10:21:38.6820123Z   --tty \
2022-12-01T10:21:38.6820358Z   --detach \
2022-12-01T10:21:38.6820625Z   --name="${container_name}" \
2022-12-01T10:21:38.6820957Z   --user jenkins \
2022-12-01T10:21:38.6821278Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-12-01T10:21:38.6821616Z   -w /var/lib/jenkins/workspace \
2022-12-01T10:21:38.6821894Z   "${DOCKER_IMAGE}"
2022-12-01T10:21:38.6822114Z )
2022-12-01T10:21:38.6822478Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-12-01T10:21:38.6834953Z shell: /usr/bin/bash -e {0}
2022-12-01T10:21:38.6835205Z env:
2022-12-01T10:21:38.6835443Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:21:38.6835691Z   GPU_FLAG: --gpus all
2022-12-01T10:21:38.6836019Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:21:38.6836336Z   PR_NUMBER: 89997
2022-12-01T10:21:38.6836569Z   BRANCH: pull/89997
2022-12-01T10:21:38.6836858Z   SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:21:38.6837188Z   BASE_SHA: ae2fe4033cf3b17259b17f351020b988fa893f91
2022-12-01T10:21:38.6837479Z   PYTORCH_RETRY_TEST_CASES: 1
2022-12-01T10:21:38.6837766Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-12-01T10:21:38.6838044Z   TEST_CONFIG: distributed
2022-12-01T10:21:38.6838295Z   SHARD_NUMBER: 2
2022-12-01T10:21:38.6838518Z   NUM_TEST_SHARDS: 3
2022-12-01T10:21:38.6839374Z   PR_BODY: Link to landed master PR (if applicable):
https://github.com/pytorch/pytorch/pull/88993

Criteria category:
1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.13
2: Prevents a crash

I'm hoping this is a low-risk change, since it's just changing one method for its safer form.
2022-12-01T10:21:38.6840263Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-12-01T10:21:38.6840594Z   SCCACHE_S3_KEY_PREFIX: pull
2022-12-01T10:21:38.6840831Z   SHM_SIZE: 2g
2022-12-01T10:21:38.6841307Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:21:38.6841777Z   XLA_CUDA: 
2022-12-01T10:21:38.6842127Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-12-01T10:21:38.6842462Z ##[endgroup]
2022-12-01T10:21:38.6871498Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-12-01T10:21:38.6871997Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2022-12-01T10:21:38.6872333Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-12-01T10:21:38.6875727Z ++ git cherry -v origin/master
2022-12-01T10:21:39.1661803Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)
2022-12-01T10:21:39.1663021Z - f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)
2022-12-01T10:21:39.1664347Z + 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)
2022-12-01T10:21:39.1665179Z - c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)
2022-12-01T10:21:39.1665843Z - 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)
2022-12-01T10:21:39.1666450Z - 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)
2022-12-01T10:21:39.1667046Z - f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)
2022-12-01T10:21:39.1667617Z - 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)
2022-12-01T10:21:39.1668232Z - 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)
2022-12-01T10:21:39.1668882Z + d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)
2022-12-01T10:21:39.1669438Z + 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)
2022-12-01T10:21:39.1670091Z - 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)
2022-12-01T10:21:39.1670549Z + f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)
2022-12-01T10:21:39.1671174Z - 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)
2022-12-01T10:21:39.1671809Z - 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)
2022-12-01T10:21:39.1672335Z + 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)
2022-12-01T10:21:39.1672988Z - aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)
2022-12-01T10:21:39.1673592Z - de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)
2022-12-01T10:21:39.1674047Z + 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)
2022-12-01T10:21:39.1674467Z + 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)
2022-12-01T10:21:39.1675108Z - f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)
2022-12-01T10:21:39.1675778Z - c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)
2022-12-01T10:21:39.1676263Z + eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)
2022-12-01T10:21:39.1676732Z + d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)
2022-12-01T10:21:39.1677319Z - 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)
2022-12-01T10:21:39.1677893Z - 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)
2022-12-01T10:21:39.1678497Z - 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)
2022-12-01T10:21:39.1678971Z + d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)
2022-12-01T10:21:39.1679591Z - 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)
2022-12-01T10:21:39.1680191Z - ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)
2022-12-01T10:21:39.1680795Z - e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)
2022-12-01T10:21:39.1681331Z + 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)
2022-12-01T10:21:39.1681949Z - 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)
2022-12-01T10:21:39.1682538Z - 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)
2022-12-01T10:21:39.1683103Z - 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)
2022-12-01T10:21:39.1683735Z - 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)
2022-12-01T10:21:39.1684380Z - 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)
2022-12-01T10:21:39.1684932Z - aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)
2022-12-01T10:21:39.1685627Z - 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)
2022-12-01T10:21:39.1686203Z + ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)
2022-12-01T10:21:39.1686904Z - d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)
2022-12-01T10:21:39.1687490Z - 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)
2022-12-01T10:21:39.1688044Z - 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)
2022-12-01T10:21:39.1688588Z - 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)
2022-12-01T10:21:39.1689022Z + 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)
2022-12-01T10:21:39.1689551Z - 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)
2022-12-01T10:21:39.1689997Z + 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)
2022-12-01T10:21:39.1690456Z + d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)
2022-12-01T10:21:39.1691176Z - d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)
2022-12-01T10:21:39.1691748Z - 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)
2022-12-01T10:21:39.1692360Z - f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)
2022-12-01T10:21:39.1693030Z - 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)
2022-12-01T10:21:39.1693514Z + 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)
2022-12-01T10:21:39.1694097Z - fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)
2022-12-01T10:21:39.1694898Z - 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)
2022-12-01T10:21:39.1695392Z - 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)
2022-12-01T10:21:39.1695946Z - 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)
2022-12-01T10:21:39.1696540Z - 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)
2022-12-01T10:21:39.1696979Z + ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)
2022-12-01T10:21:39.1697496Z - c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:21:39.1728664Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:21:39.1737541Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashI'\''m hoping this is a low-risk change, since it'\''s just changing one method for its safer form.'
2022-12-01T10:21:39.1755565Z + export 'COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:21:39.1778007Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:21:39.1786493Z + export 'PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.'
2022-12-01T10:21:39.1788311Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.'
2022-12-01T10:21:39.1789139Z +++ nproc --ignore=2
2022-12-01T10:21:39.1794825Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3591403534 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:21:52.9470841Z + container_name=76d0c0aaee307e2ea54f34b360ef1b6ca5de6826777d6857a4e0a84ac40c4542
2022-12-01T10:21:52.9475312Z ++ echo dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl
2022-12-01T10:21:52.9477466Z + docker exec -t 76d0c0aaee307e2ea54f34b360ef1b6ca5de6826777d6857a4e0a84ac40c4542 sh -c 'pip install dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-12-01T10:21:53.5282847Z Processing ./dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl
2022-12-01T10:21:53.6255902Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.13.0a0+gitc13d400) (4.3.0)
2022-12-01T10:21:53.6269512Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==1.13.0a0+gitc13d400) (3.3.0)
2022-12-01T10:21:53.6350341Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==1.13.0a0+gitc13d400) (1.21.2)
2022-12-01T10:21:54.5863653Z Installing collected packages: torch
2022-12-01T10:22:05.3220601Z Successfully installed torch-1.13.0a0+gitc13d400
2022-12-01T10:22:05.4889792Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-12-01T10:22:05.5110880Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-12-01T10:22:05.5113054Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-12-01T10:22:05.5113602Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-12-01T10:22:05.5114078Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-12-01T10:22:05.5114408Z + BUILD_DIR=build
2022-12-01T10:22:05.5118904Z + BUILD_RENAMED_DIR=build_renamed
2022-12-01T10:22:05.5119185Z + BUILD_BIN_DIR=build/bin
2022-12-01T10:22:05.5119434Z + export VALGRIND=ON
2022-12-01T10:22:05.5119678Z + VALGRIND=ON
2022-12-01T10:22:05.5120084Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]]
2022-12-01T10:22:05.5120524Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2022-12-01T10:22:05.5120844Z ++ realpath build/custom_test_artifacts
2022-12-01T10:22:05.5125399Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-12-01T10:22:05.5129025Z ++ dirname .jenkins/pytorch/test.sh
2022-12-01T10:22:05.5136649Z + source .jenkins/pytorch/common.sh
2022-12-01T10:22:05.5140725Z +++ dirname .jenkins/pytorch/common.sh
2022-12-01T10:22:05.5151382Z ++ source .jenkins/pytorch/common_utils.sh
2022-12-01T10:22:05.5153344Z +++ declare -f -t trap_add
2022-12-01T10:22:05.5157722Z ++ set -ex
2022-12-01T10:22:05.5158070Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-12-01T10:22:05.5158668Z ++ BUILD_TEST_LIBTORCH=0
2022-12-01T10:22:05.5159428Z ++ [[ distributed == *xla* ]]
2022-12-01T10:22:05.5159996Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-12-01T10:22:05.5160433Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *linux-bionic* ]]
2022-12-01T10:22:05.5160753Z ++ which conda
2022-12-01T10:22:05.5169143Z /opt/conda/bin/conda
2022-12-01T10:22:05.5170102Z ++ conda install -q -y cmake
2022-12-01T10:22:09.1093313Z Collecting package metadata (current_repodata.json): ...working... done
2022-12-01T10:22:10.4053869Z Solving environment: ...working... done
2022-12-01T10:22:10.5084204Z 
2022-12-01T10:22:10.5084425Z ## Package Plan ##
2022-12-01T10:22:10.5084660Z 
2022-12-01T10:22:10.5084796Z   environment location: /opt/conda
2022-12-01T10:22:10.5084956Z 
2022-12-01T10:22:10.5088633Z   added / updated specs:
2022-12-01T10:22:10.5089135Z     - cmake
2022-12-01T10:22:10.5089284Z 
2022-12-01T10:22:10.5089284Z 
2022-12-01T10:22:10.5089454Z The following packages will be downloaded:
2022-12-01T10:22:10.5089643Z 
2022-12-01T10:22:10.5089803Z     package                    |            build
2022-12-01T10:22:10.5090191Z     ---------------------------|-----------------
2022-12-01T10:22:10.5090618Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-12-01T10:22:10.5091049Z     ca-certificates-2022.10.11 |       h06a4308_0         124 KB
2022-12-01T10:22:10.5091471Z     certifi-2022.9.24          |  py310h06a4308_0         154 KB
2022-12-01T10:22:10.5091855Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-12-01T10:22:10.5092225Z     expat-2.4.9                |       h6a678d5_0         156 KB
2022-12-01T10:22:10.5092592Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-12-01T10:22:10.5093195Z     libcurl-7.84.0             |       h91b91d3_0         337 KB
2022-12-01T10:22:10.5093616Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-12-01T10:22:10.5093937Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-12-01T10:22:10.5094737Z     libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
2022-12-01T10:22:10.5095149Z     libgomp-11.2.0             |       h1234567_1         474 KB
2022-12-01T10:22:10.5095550Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-12-01T10:22:10.5096146Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-12-01T10:22:10.5096574Z     libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
2022-12-01T10:22:10.5097032Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-12-01T10:22:10.5097366Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-12-01T10:22:10.5097749Z     openssl-1.1.1s             |       h7f8727e_0         3.6 MB
2022-12-01T10:22:10.5098162Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-12-01T10:22:10.5098514Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-12-01T10:22:10.5098915Z     ------------------------------------------------------------
2022-12-01T10:22:10.5099258Z                                            Total:        26.3 MB
2022-12-01T10:22:10.5099436Z 
2022-12-01T10:22:10.5099546Z The following NEW packages will be INSTALLED:
2022-12-01T10:22:10.5099750Z 
2022-12-01T10:22:10.5100132Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0 None
2022-12-01T10:22:10.5100651Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0 None
2022-12-01T10:22:10.5101102Z   expat              pkgs/main/linux-64::expat-2.4.9-h6a678d5_0 None
2022-12-01T10:22:10.5101587Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0 None
2022-12-01T10:22:10.5102058Z   libcurl            pkgs/main/linux-64::libcurl-7.84.0-h91b91d3_0 None
2022-12-01T10:22:10.5102577Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0 None
2022-12-01T10:22:10.5103116Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1 None
2022-12-01T10:22:10.5103560Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0 None
2022-12-01T10:22:10.5104085Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0 None
2022-12-01T10:22:10.5104570Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0 None
2022-12-01T10:22:10.5105048Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1 None
2022-12-01T10:22:10.5105526Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1 None
2022-12-01T10:22:10.5105984Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0 None
2022-12-01T10:22:10.5106196Z 
2022-12-01T10:22:10.5106342Z The following packages will be UPDATED:
2022-12-01T10:22:10.5106532Z 
2022-12-01T10:22:10.5106869Z   ca-certificates                     2022.07.19-h06a4308_0 --> 2022.10.11-h06a4308_0 None
2022-12-01T10:22:10.5107287Z   certifi                         2022.9.14-py310h06a4308_0 --> 2022.9.24-py310h06a4308_0 None
2022-12-01T10:22:10.5107750Z   libgcc-ng                               9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:22:10.5108205Z   libgomp                                 9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:22:10.5108635Z   libstdcxx-ng                            9.3.0-hd4cf53a_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:22:10.5109088Z   openssl                                 1.1.1q-h7f8727e_0 --> 1.1.1s-h7f8727e_0 None
2022-12-01T10:22:10.5109330Z 
2022-12-01T10:22:10.5109330Z 
2022-12-01T10:22:11.9530577Z Preparing transaction: ...working... done
2022-12-01T10:22:12.5716326Z Verifying transaction: ...working... done
2022-12-01T10:22:13.6083874Z Executing transaction: ...working... done
2022-12-01T10:22:13.7070575Z Retrieving notices: ...working... done
2022-12-01T10:22:13.9122165Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-12-01T10:22:13.9122561Z + echo 'Environment variables'
2022-12-01T10:22:13.9123143Z Environment variables
2022-12-01T10:22:13.9123346Z + env
2022-12-01T10:22:13.9131103Z SHARD_NUMBER=2
2022-12-01T10:22:13.9131556Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2022-12-01T10:22:13.9132086Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2022-12-01T10:22:13.9132459Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-12-01T10:22:13.9133236Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2022-12-01T10:22:13.9133483Z UCC_HOME=/usr
2022-12-01T10:22:13.9133878Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:22:13.9134644Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2022-12-01T10:22:13.9134962Z INSTALLED_DB=yes
2022-12-01T10:22:13.9135215Z HOSTNAME=76d0c0aaee30
2022-12-01T10:22:13.9135486Z GITHUB_REF_NAME=89997/merge
2022-12-01T10:22:13.9135712Z GITHUB_API_URL=https://api.github.com
2022-12-01T10:22:13.9136018Z OPENSSL_DIR=/opt/openssl
2022-12-01T10:22:13.9136650Z UCC_COMMIT=12944da33f911daf505d9bbc51411233d0ed85e1
2022-12-01T10:22:13.9137767Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ff31eb4b-c7f2-4364-adbc-e2e6e3839c40
2022-12-01T10:22:13.9138516Z CUDA_PATH=/usr/local/cuda
2022-12-01T10:22:13.9139332Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-12-01T10:22:13.9139990Z GITHUB_RUN_ATTEMPT=1
2022-12-01T10:22:13.9140442Z TEST_CONFIG=distributed
2022-12-01T10:22:13.9141016Z NV_LIBNPP_VERSION=11.6.3.124-1
2022-12-01T10:22:13.9141485Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2022-12-01T10:22:13.9141771Z GITHUB_REPOSITORY_OWNER=pytorch
2022-12-01T10:22:13.9142041Z GITHUB_ACTIONS=true
2022-12-01T10:22:13.9142302Z NVIDIA_VISIBLE_DEVICES=all
2022-12-01T10:22:13.9142594Z NV_NVPROF_VERSION=11.6.124-1
2022-12-01T10:22:13.9142890Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2022-12-01T10:22:13.9143163Z CI=true
2022-12-01T10:22:13.9143402Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-12-01T10:22:13.9143796Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2022-12-01T10:22:13.9144105Z BRANCH=pull/89997
2022-12-01T10:22:13.9144433Z GITHUB_HEAD_REF=release/1.13-callstack
2022-12-01T10:22:13.9144752Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-12-01T10:22:13.9145112Z GITHUB_ACTOR=charlie-wt
2022-12-01T10:22:13.9145419Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-12-01T10:22:13.9145699Z GITHUB_ACTION_REF=
2022-12-01T10:22:13.9145969Z NCCL_VERSION=2.12.10-1
2022-12-01T10:22:13.9146224Z GITHUB_ACTION=__self
2022-12-01T10:22:13.9146449Z VALGRIND=ON
2022-12-01T10:22:13.9146701Z GITHUB_REF_PROTECTED=false
2022-12-01T10:22:13.9147152Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-12-01T10:22:13.9147848Z ***
2022-12-01T10:22:13.9148087Z INSTALLED_VISION=yes
2022-12-01T10:22:13.9148333Z NVARCH=x86_64
2022-12-01T10:22:13.9148622Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2022-12-01T10:22:13.9148908Z HOME=/var/lib/jenkins
2022-12-01T10:22:13.9149453Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ff31eb4b-c7f2-4364-adbc-e2e6e3839c40
2022-12-01T10:22:13.9149877Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-12-01T10:22:13.9150141Z GITHUB_ACTION_REPOSITORY=
2022-12-01T10:22:13.9150417Z GITHUB_REF_TYPE=branch
2022-12-01T10:22:13.9150731Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2022-12-01T10:22:13.9150998Z GITHUB_RETENTION_DAYS=90
2022-12-01T10:22:13.9151383Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-12-01T10:22:13.9151795Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2022-12-01T10:22:13.9152335Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ff31eb4b-c7f2-4364-adbc-e2e6e3839c40
2022-12-01T10:22:13.9152746Z DEBIAN_FRONTEND=noninteractive
2022-12-01T10:22:13.9153096Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-12-01T10:22:13.9153379Z GITHUB_REF=refs/pull/89997/merge
2022-12-01T10:22:13.9153685Z NV_CUDA_LIB_VERSION=11.6.2-1
2022-12-01T10:22:13.9153995Z GITHUB_SHA=cdc064133ad5e5a46a756ee9218659e8f252e950
2022-12-01T10:22:13.9154411Z INSTALLED_PROTOBUF=yes
2022-12-01T10:22:13.9154695Z GITHUB_RUN_ID=3591403534
2022-12-01T10:22:13.9155046Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2022-12-01T10:22:13.9155354Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-12-01T10:22:13.9155717Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-12-01T10:22:13.9156033Z NV_NVTX_VERSION=11.6.124-1
2022-12-01T10:22:13.9156334Z GITHUB_SERVER_URL=https://github.com
2022-12-01T10:22:13.9156593Z MAX_JOBS=30
2022-12-01T10:22:13.9156876Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2022-12-01T10:22:13.9157250Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2022-12-01T10:22:13.9157723Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-12-01T10:22:13.9158069Z UCX_HOME=/usr
2022-12-01T10:22:13.9158326Z PYTORCH_RETRY_TEST_CASES=1
2022-12-01T10:22:13.9158642Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-12-01T10:22:13.9158998Z BASE_SHA=ae2fe4033cf3b17259b17f351020b988fa893f91
2022-12-01T10:22:13.9159343Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2022-12-01T10:22:13.9160599Z PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.
2022-12-01T10:22:13.9161532Z GITHUB_BASE_REF=release/1.13
2022-12-01T10:22:13.9161771Z TERM=xterm
2022-12-01T10:22:13.9161995Z XLA_CUDA=
2022-12-01T10:22:13.9162270Z NV_NVML_DEV_VERSION=11.6.55-1
2022-12-01T10:22:13.9162528Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-12-01T10:22:13.9162789Z CUDA_VERSION=11.6.2
2022-12-01T10:22:13.9163130Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2022-12-01T10:22:13.9163413Z OPENSSL_ROOT_DIR=/opt/openssl
2022-12-01T10:22:13.9163971Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ff31eb4b-c7f2-4364-adbc-e2e6e3839c40
2022-12-01T10:22:13.9164373Z GITHUB_JOB=test
2022-12-01T10:22:13.9164614Z SCCACHE_S3_KEY_PREFIX=pull
2022-12-01T10:22:13.9180339Z COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack
2022-12-01T10:22:13.9187879Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-12-01T10:22:13.9188180Z NUM_TEST_SHARDS=3
2022-12-01T10:22:13.9188427Z PR_NUMBER=89997
2022-12-01T10:22:13.9188973Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ff31eb4b-c7f2-4364-adbc-e2e6e3839c40
2022-12-01T10:22:13.9189363Z SHLVL=1
2022-12-01T10:22:13.9189709Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2022-12-01T10:22:13.9190040Z GITHUB_REPOSITORY=pytorch/pytorch
2022-12-01T10:22:13.9190633Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471
2022-12-01T10:22:13.9191224Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2022-12-01T10:22:13.9191545Z SHA1=c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:22:13.9191834Z GITHUB_EVENT_NAME=pull_request
2022-12-01T10:22:13.9192148Z NV_CUDA_CUDART_VERSION=11.6.55-1
2022-12-01T10:22:13.9192504Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-12-01T10:22:13.9192795Z GITHUB_RUN_NUMBER=69319
2022-12-01T10:22:13.9193038Z GITHUB_WORKFLOW=pull
2022-12-01T10:22:13.9193457Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:22:13.9193912Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2022-12-01T10:22:13.9194336Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:22:13.9194740Z GITHUB_TRIGGERING_ACTOR=charlie-wt
2022-12-01T10:22:13.9195010Z _=/usr/bin/env
2022-12-01T10:22:13.9195277Z + echo 'Testing pytorch'
2022-12-01T10:22:13.9195536Z Testing pytorch
2022-12-01T10:22:13.9195809Z + export LANG=C.UTF-8
2022-12-01T10:22:13.9196052Z + LANG=C.UTF-8
2022-12-01T10:22:13.9196322Z + PR_NUMBER=89997
2022-12-01T10:22:13.9196591Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-12-01T10:22:13.9196875Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-12-01T10:22:13.9197280Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-12-01T10:22:13.9197602Z + [[ distributed == \s\l\o\w ]]
2022-12-01T10:22:13.9198015Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2022-12-01T10:22:13.9198442Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-12-01T10:22:13.9198792Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-12-01T10:22:13.9199112Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-12-01T10:22:13.9199504Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda11* ]]
2022-12-01T10:22:13.9199826Z + export BUILD_SPLIT_CUDA=ON
2022-12-01T10:22:13.9200096Z + BUILD_SPLIT_CUDA=ON
2022-12-01T10:22:13.9200354Z + [[ distributed == *crossref* ]]
2022-12-01T10:22:13.9200637Z + [[ distributed == *dynamo* ]]
2022-12-01T10:22:13.9200928Z + [[ -n 89997 ]]
2022-12-01T10:22:13.9201172Z + [[ -z '' ]]
2022-12-01T10:22:13.9201461Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-12-01T10:22:13.9201791Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-12-01T10:22:13.9202178Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-12-01T10:22:13.9202617Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2022-12-01T10:22:13.9202979Z + pip_install --user ninja
2022-12-01T10:22:13.9203323Z + pip install --progress-bar off --user ninja
2022-12-01T10:22:14.4961980Z Collecting ninja
2022-12-01T10:22:14.5673244Z   Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)
2022-12-01T10:22:15.4628174Z Installing collected packages: ninja
2022-12-01T10:22:15.4739567Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-12-01T10:22:15.4740427Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T10:22:15.4791860Z Successfully installed ninja-1.11.1
2022-12-01T10:22:15.5525596Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:22:15.5526283Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:22:15.5526844Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2022-12-01T10:22:15.5527201Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-12-01T10:22:15.5527519Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-12-01T10:22:15.5531684Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2022-12-01T10:22:15.5547231Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-12-01T10:22:15.5547671Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2022-12-01T10:22:15.5550855Z + cd test
2022-12-01T10:22:15.5551265Z + python -c 'import torch; print(torch.__config__.show())'
2022-12-01T10:22:16.8410122Z PyTorch built with:
2022-12-01T10:22:16.8410582Z   - GCC 7.5
2022-12-01T10:22:16.8410909Z   - C++ Version: 201402
2022-12-01T10:22:16.8411458Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-12-01T10:22:16.8411996Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-12-01T10:22:16.8412412Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-12-01T10:22:16.8412797Z   - LAPACK is enabled (usually provided by MKL)
2022-12-01T10:22:16.8413109Z   - NNPACK is enabled
2022-12-01T10:22:16.8413421Z   - CPU capability usage: AVX2
2022-12-01T10:22:16.8413727Z   - CUDA Runtime 11.6
2022-12-01T10:22:16.8414217Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-12-01T10:22:16.8414881Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-12-01T10:22:16.8415184Z   - Magma 2.6.1
2022-12-01T10:22:16.8418177Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-12-01T10:22:16.8420315Z 
2022-12-01T10:22:17.0300963Z + cd test
2022-12-01T10:22:17.0302037Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-12-01T10:22:18.2291730Z ATen/Parallel:
2022-12-01T10:22:18.2292084Z 	at::get_num_threads() : 16
2022-12-01T10:22:18.2292392Z 	at::get_num_interop_threads() : 16
2022-12-01T10:22:18.2292687Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-12-01T10:22:18.2292952Z 	omp_get_max_threads() : 16
2022-12-01T10:22:18.2293580Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-12-01T10:22:18.2293977Z 	mkl_get_max_threads() : 16
2022-12-01T10:22:18.2295059Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-12-01T10:22:18.2295475Z std::thread::hardware_concurrency() : 32
2022-12-01T10:22:18.2295902Z Environment variables:
2022-12-01T10:22:18.2296141Z 	OMP_NUM_THREADS : [not set]
2022-12-01T10:22:18.2296423Z 	MKL_NUM_THREADS : [not set]
2022-12-01T10:22:18.2296703Z ATen parallel backend: OpenMP
2022-12-01T10:22:18.2296883Z 
2022-12-01T10:22:18.4079290Z + [[ distributed == *backward* ]]
2022-12-01T10:22:18.4079617Z + [[ distributed == *xla* ]]
2022-12-01T10:22:18.4079934Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-12-01T10:22:18.4080447Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-12-01T10:22:18.4080798Z + [[ distributed == distributed ]]
2022-12-01T10:22:18.4081056Z + install_torchdynamo
2022-12-01T10:22:18.4081299Z + local commit
2022-12-01T10:22:18.4083895Z ++ get_pinned_commit torchdynamo
2022-12-01T10:22:18.4084219Z ++ cat .github/ci_commit_pins/torchdynamo.txt
2022-12-01T10:22:18.4100487Z + commit=6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:22:18.4101129Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:22:18.4101849Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:22:18.9192225Z Collecting git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:22:18.9197399Z   Cloning https://github.com/pytorch/torchdynamo.git (to revision 6ead5cae0d1234aa64db06fe230ef56e12ec76fe) to /tmp/pip-req-build-gtmbrcas
2022-12-01T10:22:18.9219979Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-gtmbrcas
2022-12-01T10:22:19.5254636Z   Running command git rev-parse -q --verify 'sha^6ead5cae0d1234aa64db06fe230ef56e12ec76fe'
2022-12-01T10:22:19.5277688Z   Running command git fetch -q https://github.com/pytorch/torchdynamo.git 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:22:19.8062748Z   Running command git checkout -q 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:22:20.4698868Z   Resolved https://github.com/pytorch/torchdynamo.git to commit 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:22:22.4961014Z   Preparing metadata (setup.py) ... [?25l- done
2022-12-01T10:22:22.5029801Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+gitc13d400)
2022-12-01T10:22:22.5079130Z Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.21.2)
2022-12-01T10:22:22.5489110Z Collecting tabulate
2022-12-01T10:22:22.6060540Z   Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
2022-12-01T10:22:22.6217476Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from torchdynamo==1.13.0.dev0) (6.0)
2022-12-01T10:22:22.6223938Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.11.1)
2022-12-01T10:22:22.6264986Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0)
2022-12-01T10:22:22.6301887Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1)
2022-12-01T10:22:22.6427521Z Building wheels for collected packages: torchdynamo
2022-12-01T10:22:26.7255760Z   Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done
2022-12-01T10:22:26.7358872Z [?25h  Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp310-cp310-linux_x86_64.whl size=2693850 sha256=0e82c3679bddf3603b4ae0a5174e019ee014fb9be4aa1e989360476a93db5712
2022-12-01T10:22:26.7360193Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/6c/b6/c8/fbfe87a581aa24ccebb62be69424794eca3c035823c5b100d9
2022-12-01T10:22:26.7384697Z Successfully built torchdynamo
2022-12-01T10:22:27.6153645Z Installing collected packages: tabulate, torchdynamo
2022-12-01T10:22:27.9797610Z Successfully installed tabulate-0.9.0 torchdynamo-1.13.0.dev0
2022-12-01T10:22:28.0669801Z + test_distributed
2022-12-01T10:22:28.0670316Z + echo 'Testing distributed python tests'
2022-12-01T10:22:28.0670636Z Testing distributed python tests
2022-12-01T10:22:28.0671017Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose
2022-12-01T10:22:29.9091843Z Ignoring disabled issues:  []
2022-12-01T10:22:29.9474806Z /var/lib/jenkins/workspace/test/run_test.py:1050: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-12-01T10:22:29.9475374Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-12-01T10:22:29.9478479Z Found test time stats from artifacts
2022-12-01T10:22:29.9489139Z Selected tests:
2022-12-01T10:22:29.9489464Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-12-01T10:22:29.9489780Z  distributed/test_c10d_gloo
2022-12-01T10:22:29.9490111Z  distributed/fsdp/test_fsdp_mixed_precision
2022-12-01T10:22:29.9490526Z  distributed/fsdp/test_fsdp_summon_full_params
2022-12-01T10:22:29.9490878Z  distributed/optim/test_zero_redundancy_optimizer
2022-12-01T10:22:29.9491132Z  distributed/fsdp/test_fsdp_comm_hooks
2022-12-01T10:22:29.9491429Z  distributed/fsdp/test_wrap
2022-12-01T10:22:29.9491736Z  distributed/algorithms/quantization/test_quantization
2022-12-01T10:22:29.9492035Z  distributed/test_store
2022-12-01T10:22:29.9492323Z  distributed/test_c10d_spawn_nccl
2022-12-01T10:22:29.9492611Z  distributed/algorithms/test_join
2022-12-01T10:22:29.9492900Z  distributed/fsdp/test_fsdp_comm
2022-12-01T10:22:29.9493192Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-12-01T10:22:29.9493526Z  distributed/test_c10d_common
2022-12-01T10:22:29.9493813Z  distributed/fsdp/test_fsdp_meta
2022-12-01T10:22:29.9494154Z  distributed/elastic/multiprocessing/api_test
2022-12-01T10:22:29.9494744Z  distributed/fsdp/test_fsdp_tp_integration
2022-12-01T10:22:29.9495135Z  distributed/fsdp/test_fsdp_memory
2022-12-01T10:22:29.9495419Z  distributed/fsdp/test_fsdp_overlap
2022-12-01T10:22:29.9495757Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-12-01T10:22:29.9496124Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-12-01T10:22:29.9496506Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-12-01T10:22:29.9496764Z  distributed/fsdp/test_fsdp_input
2022-12-01T10:22:29.9497068Z  distributed/_shard/sharded_tensor/ops/test_init
2022-12-01T10:22:29.9497418Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-12-01T10:22:29.9497725Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-12-01T10:22:29.9498074Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-12-01T10:22:29.9498407Z  distributed/fsdp/test_fsdp_pure_fp16
2022-12-01T10:22:29.9498714Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-12-01T10:22:29.9499049Z  distributed/fsdp/test_fsdp_traversal
2022-12-01T10:22:29.9499366Z  distributed/test_data_parallel
2022-12-01T10:22:29.9499657Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-12-01T10:22:29.9500031Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-12-01T10:22:29.9500377Z  distributed/elastic/utils/logging_test
2022-12-01T10:22:29.9500688Z  distributed/elastic/metrics/api_test
2022-12-01T10:22:29.9500960Z  distributed/fsdp/test_utils
2022-12-01T10:22:29.9501230Z  distributed/test_nccl
2022-12-01T10:22:29.9501556Z  distributed/_shard/checkpoint/test_file_system_checkpoint_cpu
2022-12-01T10:22:29.9501909Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-12-01T10:22:29.9502234Z  distributed/elastic/events/lib_test
2022-12-01T10:22:29.9502564Z  distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-12-01T10:22:29.9502885Z  distributed/pipeline/sync/skip/test_gpipe
2022-12-01T10:22:29.9503216Z  distributed/pipeline/sync/skip/test_portal
2022-12-01T10:22:29.9503570Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-12-01T10:22:29.9504100Z  distributed/pipeline/sync/test_checkpoint
2022-12-01T10:22:29.9504453Z  distributed/pipeline/sync/test_dependency
2022-12-01T10:22:29.9504862Z  distributed/pipeline/sync/test_phony
2022-12-01T10:22:29.9505156Z  distributed/pipeline/sync/test_stream
2022-12-01T10:22:29.9505463Z  distributed/rpc/test_faulty_agent
2022-12-01T10:22:30.1514112Z Prioritized test from test file changes.
2022-12-01T10:22:30.1514402Z reordering tests for PR:
2022-12-01T10:22:30.1516741Z prioritized: ['distributed/test_c10d_gloo', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/fsdp/test_wrap', 'distributed/test_store', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_sharded_grad_scaler', 'distributed/test_c10d_common', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_overlap', 'distributed/fsdp/test_fsdp_input', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_utils', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu', 'distributed/fsdp/test_fsdp_param_exec_order_wrap']
2022-12-01T10:22:30.1520643Z the rest: ['distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/optim/test_zero_redundancy_optimizer', 'distributed/algorithms/quantization/test_quantization', 'distributed/test_c10d_spawn_nccl', 'distributed/algorithms/test_join', 'distributed/elastic/multiprocessing/api_test', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/test_data_parallel', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/elastic/utils/logging_test', 'distributed/elastic/metrics/api_test', 'distributed/test_nccl', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/elastic/events/lib_test', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_stream', 'distributed/rpc/test_faulty_agent']
2022-12-01T10:22:30.1522587Z 
2022-12-01T10:22:30.1523083Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-12-01T10:22:30.1729284Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-12-01T10:22:30.1892359Z parallel (file granularity) tests:
2022-12-01T10:22:30.1892660Z  
2022-12-01T10:22:30.1892908Z serial (file granularity) tests:
2022-12-01T10:22:30.1893160Z  distributed/test_c10d_gloo
2022-12-01T10:22:30.1893477Z  distributed/fsdp/test_fsdp_mixed_precision
2022-12-01T10:22:30.1893856Z  distributed/fsdp/test_fsdp_summon_full_params
2022-12-01T10:22:30.1894152Z  distributed/fsdp/test_fsdp_comm_hooks
2022-12-01T10:22:30.1894748Z  distributed/fsdp/test_wrap
2022-12-01T10:22:30.1894995Z  distributed/test_store
2022-12-01T10:22:30.1895261Z  distributed/fsdp/test_fsdp_comm
2022-12-01T10:22:30.1895584Z  distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-12-01T10:22:30.1895886Z  distributed/test_c10d_common
2022-12-01T10:22:30.1896119Z  distributed/fsdp/test_fsdp_meta
2022-12-01T10:22:30.1896429Z  distributed/fsdp/test_fsdp_tp_integration
2022-12-01T10:22:30.1896775Z  distributed/fsdp/test_fsdp_memory
2022-12-01T10:22:30.1897195Z  distributed/fsdp/test_fsdp_overlap
2022-12-01T10:22:30.1897518Z  distributed/fsdp/test_fsdp_input
2022-12-01T10:22:30.1897848Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-12-01T10:22:30.1898285Z  distributed/fsdp/test_fsdp_pure_fp16
2022-12-01T10:22:30.1898592Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-12-01T10:22:30.1898910Z  distributed/fsdp/test_fsdp_traversal
2022-12-01T10:22:30.1899216Z  distributed/fsdp/test_utils
2022-12-01T10:22:30.1899529Z  distributed/_shard/checkpoint/test_file_system_checkpoint_cpu
2022-12-01T10:22:30.1899898Z  distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-12-01T10:22:30.1900236Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-12-01T10:22:30.1900570Z  distributed/optim/test_zero_redundancy_optimizer
2022-12-01T10:22:30.1900937Z  distributed/algorithms/quantization/test_quantization
2022-12-01T10:22:30.1901263Z  distributed/test_c10d_spawn_nccl
2022-12-01T10:22:30.1901554Z  distributed/algorithms/test_join
2022-12-01T10:22:30.1901862Z  distributed/elastic/multiprocessing/api_test
2022-12-01T10:22:30.1902213Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-12-01T10:22:30.1902574Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-12-01T10:22:30.1902942Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-12-01T10:22:30.1903287Z  distributed/_shard/sharded_tensor/ops/test_init
2022-12-01T10:22:30.1903627Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-12-01T10:22:30.1903959Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-12-01T10:22:30.1904277Z  distributed/test_data_parallel
2022-12-01T10:22:30.1904600Z  distributed/_shard/sharded_optim/test_sharded_optim
2022-12-01T10:22:30.1904947Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-12-01T10:22:30.1905295Z  distributed/elastic/utils/logging_test
2022-12-01T10:22:30.1905594Z  distributed/elastic/metrics/api_test
2022-12-01T10:22:30.1905858Z  distributed/test_nccl
2022-12-01T10:22:30.1906174Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2022-12-01T10:22:30.1906501Z  distributed/elastic/events/lib_test
2022-12-01T10:22:30.1906815Z  distributed/pipeline/sync/skip/test_gpipe
2022-12-01T10:22:30.1907134Z  distributed/pipeline/sync/skip/test_portal
2022-12-01T10:22:30.1907483Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-12-01T10:22:30.1907828Z  distributed/pipeline/sync/test_checkpoint
2022-12-01T10:22:30.1908134Z  distributed/pipeline/sync/test_dependency
2022-12-01T10:22:30.1908450Z  distributed/pipeline/sync/test_phony
2022-12-01T10:22:30.1908759Z  distributed/pipeline/sync/test_stream
2022-12-01T10:22:30.1909044Z  distributed/rpc/test_faulty_agent
2022-12-01T10:22:31.9375613Z Ignoring disabled issues:  []
2022-12-01T10:22:31.9847562Z Ignoring disabled issues:  []
2022-12-01T10:22:32.2959794Z Running distributed/test_c10d_gloo ... [2022-12-01 10:22:32.295495]
2022-12-01T10:22:32.2962207Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:22:32.295922]
2022-12-01T10:35:33.4620371Z 
2022-12-01T10:35:33.4620871Z Expand the folded group to see the log file of distributed/test_c10d_gloo
2022-12-01T10:35:33.4628283Z ##[group]PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_8pmfbaxw)
2022-12-01T10:35:33.4630257Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2022-12-01T10:35:33.4632248Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2022-12-01T10:35:33.4632798Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2022-12-01T10:35:33.4633123Z test_gloo_barrier_device_ids (__main__.CommTest)
2022-12-01T10:35:33.4633536Z test_gloo_rank_membership (__main__.CommTest)
2022-12-01T10:35:33.4633808Z test_gloo_warn_not_in_group (__main__.CommTest)
2022-12-01T10:35:33.4634208Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2022-12-01T10:35:33.4634546Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2022-12-01T10:35:33.4634896Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2022-12-01T10:35:33.4635196Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2022-12-01T10:35:33.4635560Z test_tensor_dtype_complex (__main__.CommTest)
2022-12-01T10:35:33.4635946Z test_tensor_dtype_mismatch (__main__.CommTest)
2022-12-01T10:35:33.4637109Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2022-12-01T10:35:33.4638293Z test_allgather_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:35:33.4638597Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:35:33.4638957Z test_allreduce_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:35:33.4639471Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:35:33.4640066Z test_broadcast_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:35:33.4642845Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:35:33.4643418Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:35:33.4643967Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:35:33.4644710Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2022-12-01T10:35:33.4645311Z test_scatter_work_wait_cpu (__main__.CompilerTest)
2022-12-01T10:35:33.4645871Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2022-12-01T10:35:33.4656621Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2022-12-01T10:35:33.4666831Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4667737Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4668634Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4669523Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4670450Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4671432Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4672289Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4673181Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4674048Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4675007Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4675909Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4676844Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4677777Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4678607Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4679367Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4680200Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4682811Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4683585Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4684428Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4685307Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4686106Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4686882Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4687772Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4688587Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4689519Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4690338Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4691081Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4691798Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4692577Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4693332Z test_ignored_output (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4694068Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4695687Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4696481Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4697177Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4697945Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4698657Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4699496Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.4700493Z <unittest.suite.TestSuite tests=[<__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>]>
2022-12-01T10:35:33.4701509Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2022-12-01T10:35:33.4702264Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:35:33.4711392Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2022-12-01T10:35:33.4718629Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4719022Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4719393Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4719759Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4720153Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4720557Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4720926Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4721298Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4721664Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4722014Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4722409Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4722826Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4723216Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4723578Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4723974Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4724364Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4724753Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4725155Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4725529Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4725880Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4726254Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4726619Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4726986Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4727396Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4727772Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4728138Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4728543Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4728893Z test_gather_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4729297Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4729639Z test_gather_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4730014Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4730389Z test_gather_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4730749Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4731112Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4731479Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4731841Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4732184Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4732528Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4732886Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4733226Z test_round_robin (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4733598Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4733967Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4734949Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4735519Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4735872Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4736236Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4736587Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4736967Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4737364Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4737743Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2022-12-01T10:35:33.4738600Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2022-12-01T10:35:33.4739390Z test_forward_backward (__main__.ReducerTest)
2022-12-01T10:35:33.4739728Z test_forward_backward_optimizer (__main__.ReducerTest)
2022-12-01T10:35:33.4740093Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2022-12-01T10:35:33.4740433Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2022-12-01T10:35:33.4740782Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2022-12-01T10:35:33.4741128Z test_single_dtype_single_bucket (__main__.ReducerTest)
2022-12-01T10:35:33.4741532Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2022-12-01T10:35:33.4741942Z test_logging_init (__main__.RendezvousEnvTest)
2022-12-01T10:35:33.4742263Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:35:33.4742664Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2022-12-01T10:35:33.4743081Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2022-12-01T10:35:33.4743777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4744227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4744786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4745352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4745595Z 
2022-12-01T10:35:33.4745705Z Running tests...
2022-12-01T10:35:33.4746100Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4746704Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4747186Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4747649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 770
2022-12-01T10:35:33.4748071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 771
2022-12-01T10:35:33.4748664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4749112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4749667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4750133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4750712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4751156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4751712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4752180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4752613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4753083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4753408Z ok (3.860s)
2022-12-01T10:35:33.4753554Z 
2022-12-01T10:35:33.4753830Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4754154Z Ran 1 test in 3.860s
2022-12-01T10:35:33.4754296Z 
2022-12-01T10:35:33.4754389Z OK
2022-12-01T10:35:33.4754524Z 
2022-12-01T10:35:33.4754647Z Generating XML reports...
2022-12-01T10:35:33.4755185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102236.xml
2022-12-01T10:35:33.4755828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4756272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4756840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4757305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4757532Z 
2022-12-01T10:35:33.4757623Z Running tests...
2022-12-01T10:35:33.4758026Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4758552Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4759018Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4759486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 879
2022-12-01T10:35:33.4759924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 880
2022-12-01T10:35:33.4760521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4760950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4761513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4761978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4762608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4763040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4763658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4764118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4764538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4765006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4765348Z ok (5.354s)
2022-12-01T10:35:33.4765493Z 
2022-12-01T10:35:33.4765758Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4766061Z Ran 1 test in 5.354s
2022-12-01T10:35:33.4766220Z 
2022-12-01T10:35:33.4766311Z OK
2022-12-01T10:35:33.4766442Z 
2022-12-01T10:35:33.4766564Z Generating XML reports...
2022-12-01T10:35:33.4767086Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102242.xml
2022-12-01T10:35:33.4767750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4768193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4768758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4769207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4769434Z 
2022-12-01T10:35:33.4769543Z Running tests...
2022-12-01T10:35:33.4769943Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4770451Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4770920Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4771370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 990
2022-12-01T10:35:33.4771807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 991
2022-12-01T10:35:33.4772398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4772842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4773410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4773857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4774815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4775325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4775902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4776345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4776782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4777255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4777722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4778214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4778874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4779559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4780033Z ok (3.753s)
2022-12-01T10:35:33.4780196Z 
2022-12-01T10:35:33.4780469Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4780790Z Ran 1 test in 3.753s
2022-12-01T10:35:33.4781013Z 
2022-12-01T10:35:33.4781105Z OK
2022-12-01T10:35:33.4781219Z 
2022-12-01T10:35:33.4781342Z Generating XML reports...
2022-12-01T10:35:33.4781881Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102249.xml
2022-12-01T10:35:33.4782532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4782959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4783527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4783992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4784218Z 
2022-12-01T10:35:33.4784329Z Running tests...
2022-12-01T10:35:33.4784714Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4785238Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4785713Z   test_gloo_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4786151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1099
2022-12-01T10:35:33.4786594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1100
2022-12-01T10:35:33.4787192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4787634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4788184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4788648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4789220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4789648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4790215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4790672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4791107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4791557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4792034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4792523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4793181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4793852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4794383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:35:33.4794869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:35:33.4795499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4796176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4796565Z ok (3.819s)
2022-12-01T10:35:33.4796710Z 
2022-12-01T10:35:33.4797031Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4797348Z Ran 1 test in 3.819s
2022-12-01T10:35:33.4797506Z 
2022-12-01T10:35:33.4797597Z OK
2022-12-01T10:35:33.4797727Z 
2022-12-01T10:35:33.4797900Z Generating XML reports...
2022-12-01T10:35:33.4798423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102255.xml
2022-12-01T10:35:33.4799078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4799522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4800094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4800546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4800775Z 
2022-12-01T10:35:33.4800884Z Running tests...
2022-12-01T10:35:33.4801282Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4801793Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4802259Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4802713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1211
2022-12-01T10:35:33.4803151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1212
2022-12-01T10:35:33.4803737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4804226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4804796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4805264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4805825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4806266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4806838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4807284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4807718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4808185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4808666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4809141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4809795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4810473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4811007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:35:33.4811476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:35:33.4812122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4812800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4813169Z ok (5.370s)
2022-12-01T10:35:33.4813316Z 
2022-12-01T10:35:33.4813585Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4813907Z Ran 1 test in 5.370s
2022-12-01T10:35:33.4814064Z 
2022-12-01T10:35:33.4814155Z OK
2022-12-01T10:35:33.4814627Z 
2022-12-01T10:35:33.4814844Z Generating XML reports...
2022-12-01T10:35:33.4815401Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102301.xml
2022-12-01T10:35:33.4816154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4816587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4817160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4817628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4817856Z 
2022-12-01T10:35:33.4817963Z Running tests...
2022-12-01T10:35:33.4818345Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4818866Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4819364Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4819824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1325
2022-12-01T10:35:33.4820269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1326
2022-12-01T10:35:33.4820866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4821309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4821859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4822322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4822896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4823337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4823888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4824351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4824784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4825238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4825720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4826213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4826869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4827539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4828068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:35:33.4828560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:35:33.4829204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4829911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4830299Z ok (5.362s)
2022-12-01T10:35:33.4830443Z 
2022-12-01T10:35:33.4830709Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4831019Z Ran 1 test in 5.362s
2022-12-01T10:35:33.4831178Z 
2022-12-01T10:35:33.4831269Z OK
2022-12-01T10:35:33.4831399Z 
2022-12-01T10:35:33.4831521Z Generating XML reports...
2022-12-01T10:35:33.4832126Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102309.xml
2022-12-01T10:35:33.4832789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4833332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4833903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4834352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4834579Z 
2022-12-01T10:35:33.4834689Z Running tests...
2022-12-01T10:35:33.4835089Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4835610Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4836091Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4836575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1442
2022-12-01T10:35:33.4837019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1443
2022-12-01T10:35:33.4837604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4838048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4838617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4839081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4839636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4840077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4840647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4841107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4841527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4841995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4842383Z skip: Need at least 4 CUDA devices (3.863s)
2022-12-01T10:35:33.4842574Z 
2022-12-01T10:35:33.4842828Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4843153Z Ran 1 test in 3.864s
2022-12-01T10:35:33.4843311Z 
2022-12-01T10:35:33.4843418Z OK (skipped=1)
2022-12-01T10:35:33.4843570Z 
2022-12-01T10:35:33.4843693Z Generating XML reports...
2022-12-01T10:35:33.4844210Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102316.xml
2022-12-01T10:35:33.4844871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4845314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4845871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4846334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4846562Z 
2022-12-01T10:35:33.4846670Z Running tests...
2022-12-01T10:35:33.4847074Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4847580Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4848063Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4848529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1545
2022-12-01T10:35:33.4849007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1546
2022-12-01T10:35:33.4849619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4850135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4850707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4851158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4851726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4852165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4852712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4853172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4853609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4854079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4855016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4855518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4856178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4856862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4857235Z ok (3.853s)
2022-12-01T10:35:33.4857381Z 
2022-12-01T10:35:33.4857652Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4857976Z Ran 1 test in 3.853s
2022-12-01T10:35:33.4858139Z 
2022-12-01T10:35:33.4858215Z OK
2022-12-01T10:35:33.4858347Z 
2022-12-01T10:35:33.4858472Z Generating XML reports...
2022-12-01T10:35:33.4859018Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102323.xml
2022-12-01T10:35:33.4859675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4860104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4860675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4861137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4861365Z 
2022-12-01T10:35:33.4861456Z Running tests...
2022-12-01T10:35:33.4861855Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4862379Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4862862Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4863314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1654
2022-12-01T10:35:33.4863753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1655
2022-12-01T10:35:33.4864356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4864785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4865354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4865818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4866394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4866905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4867489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4868015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4868431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4868898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4869376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4869863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4870502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4871189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4871717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:35:33.4872210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:35:33.4872836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4873514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:35:33.4873904Z ok (3.858s)
2022-12-01T10:35:33.4874049Z 
2022-12-01T10:35:33.4874316Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4874622Z Ran 1 test in 3.858s
2022-12-01T10:35:33.4874781Z 
2022-12-01T10:35:33.4874873Z OK
2022-12-01T10:35:33.4875004Z 
2022-12-01T10:35:33.4875130Z Generating XML reports...
2022-12-01T10:35:33.4875652Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102329.xml
2022-12-01T10:35:33.4876311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4876762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4877330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4877780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4878004Z 
2022-12-01T10:35:33.4878112Z Running tests...
2022-12-01T10:35:33.4878516Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4879023Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4879495Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4879947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1769
2022-12-01T10:35:33.4880396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1770
2022-12-01T10:35:33.4880979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4881423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4881988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4882435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4883007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4883447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4884078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4884531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4885012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4885480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4885946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4886435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4887088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4887770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4888147Z ok (3.850s)
2022-12-01T10:35:33.4888292Z 
2022-12-01T10:35:33.4888561Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4888888Z Ran 1 test in 3.851s
2022-12-01T10:35:33.4889046Z 
2022-12-01T10:35:33.4889120Z OK
2022-12-01T10:35:33.4889251Z 
2022-12-01T10:35:33.4889374Z Generating XML reports...
2022-12-01T10:35:33.4889908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102335.xml
2022-12-01T10:35:33.4890566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4890992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4891560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4892023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4892250Z 
2022-12-01T10:35:33.4892362Z Running tests...
2022-12-01T10:35:33.4892744Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4893269Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4893738Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4894177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1878
2022-12-01T10:35:33.4894872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1879
2022-12-01T10:35:33.4895484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4895929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4896481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4896953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4897529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4897956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4898525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4898984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4899421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4899876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4900348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4900936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4901586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4902339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4903377Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.4903999Z   warnings.warn(
2022-12-01T10:35:33.4904861Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.4905451Z   warnings.warn(
2022-12-01T10:35:33.4906314Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.4906928Z   warnings.warn(
2022-12-01T10:35:33.4907778Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.4908371Z   warnings.warn(
2022-12-01T10:35:33.4908604Z ok (3.847s)
2022-12-01T10:35:33.4908749Z 
2022-12-01T10:35:33.4909022Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4909330Z Ran 1 test in 3.847s
2022-12-01T10:35:33.4909488Z 
2022-12-01T10:35:33.4909584Z OK
2022-12-01T10:35:33.4909716Z 
2022-12-01T10:35:33.4909837Z Generating XML reports...
2022-12-01T10:35:33.4910368Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102341.xml
2022-12-01T10:35:33.4911014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4911459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4912027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4912494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4912703Z 
2022-12-01T10:35:33.4912810Z Running tests...
2022-12-01T10:35:33.4913210Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4913731Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4914197Z   test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4914659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1987
2022-12-01T10:35:33.4915104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1988
2022-12-01T10:35:33.4915705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4916133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4916703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4917167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4917723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4918219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4918799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4919312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4919726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4920193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4920677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4921152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4921808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4922495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4923412Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4924131Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4924963Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4925671Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4925993Z ok (3.872s)
2022-12-01T10:35:33.4926139Z 
2022-12-01T10:35:33.4926387Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4926716Z Ran 1 test in 3.872s
2022-12-01T10:35:33.4926874Z 
2022-12-01T10:35:33.4926966Z OK
2022-12-01T10:35:33.4927097Z 
2022-12-01T10:35:33.4927222Z Generating XML reports...
2022-12-01T10:35:33.4927760Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102347.xml
2022-12-01T10:35:33.4928429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4928874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4929474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4929940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4930169Z 
2022-12-01T10:35:33.4930276Z Running tests...
2022-12-01T10:35:33.4930673Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4931183Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4931664Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4932129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2096
2022-12-01T10:35:33.4932551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2097
2022-12-01T10:35:33.4933152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4933597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4934166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4934837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4935508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4935961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4936621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4937067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4937502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4937970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4938439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4938930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4939589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4940275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4941233Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4941946Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4942781Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4943484Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4943791Z ok (5.462s)
2022-12-01T10:35:33.4943940Z 
2022-12-01T10:35:33.4944212Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4944537Z Ran 1 test in 5.462s
2022-12-01T10:35:33.4944697Z 
2022-12-01T10:35:33.4944789Z OK
2022-12-01T10:35:33.4944902Z 
2022-12-01T10:35:33.4945024Z Generating XML reports...
2022-12-01T10:35:33.4945571Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102353.xml
2022-12-01T10:35:33.4946234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4946663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4947232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4947697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4947926Z 
2022-12-01T10:35:33.4948034Z Running tests...
2022-12-01T10:35:33.4948420Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4948940Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4949426Z   test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4949874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2207
2022-12-01T10:35:33.4950315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2208
2022-12-01T10:35:33.4950913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4951357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4951909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4952432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4953018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4953512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4954063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4954525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4954957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4955409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4955886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4956378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4957033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4957707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4958626Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4959339Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4960176Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4960867Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4961702Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4962402Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4963232Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4963934Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4964245Z ok (3.868s)
2022-12-01T10:35:33.4964391Z 
2022-12-01T10:35:33.4964660Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4964989Z Ran 1 test in 3.868s
2022-12-01T10:35:33.4965148Z 
2022-12-01T10:35:33.4965222Z OK
2022-12-01T10:35:33.4965353Z 
2022-12-01T10:35:33.4965474Z Generating XML reports...
2022-12-01T10:35:33.4966030Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102401.xml
2022-12-01T10:35:33.4966703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4967134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4967701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4968163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4968391Z 
2022-12-01T10:35:33.4968482Z Running tests...
2022-12-01T10:35:33.4968882Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4969459Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4969947Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4970444Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2316
2022-12-01T10:35:33.4970885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2317
2022-12-01T10:35:33.4971489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4971918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4972486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4972954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4973533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4973958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4974752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4975230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4975648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4976119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4976598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4977086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4977731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4978423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4979350Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4980063Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4980897Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4981603Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4982448Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4983157Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4983985Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.4984665Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.4984985Z ok (5.362s)
2022-12-01T10:35:33.4985135Z 
2022-12-01T10:35:33.4985401Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4985708Z Ran 1 test in 5.362s
2022-12-01T10:35:33.4985865Z 
2022-12-01T10:35:33.4985957Z OK
2022-12-01T10:35:33.4986087Z 
2022-12-01T10:35:33.4986293Z Generating XML reports...
2022-12-01T10:35:33.4986860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102407.xml
2022-12-01T10:35:33.4987580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4988024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4988592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4989056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4989267Z 
2022-12-01T10:35:33.4989374Z Running tests...
2022-12-01T10:35:33.4989767Z ----------------------------------------------------------------------
2022-12-01T10:35:33.4990291Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.4990760Z   test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.4991224Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2427
2022-12-01T10:35:33.4991667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2428
2022-12-01T10:35:33.4992268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4992696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4993265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4993725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4994277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.4994720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.4995291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.4995752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.4996167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.4996639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.4997124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.4997601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.4998260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4998947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.4999866Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5000585Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5001409Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5002108Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5002431Z ok (3.848s)
2022-12-01T10:35:33.5002575Z 
2022-12-01T10:35:33.5002826Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5003206Z Ran 1 test in 3.848s
2022-12-01T10:35:33.5003372Z 
2022-12-01T10:35:33.5003464Z OK
2022-12-01T10:35:33.5003595Z 
2022-12-01T10:35:33.5003717Z Generating XML reports...
2022-12-01T10:35:33.5004305Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102414.xml
2022-12-01T10:35:33.5004970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5005406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5005986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5006455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5006666Z 
2022-12-01T10:35:33.5006774Z Running tests...
2022-12-01T10:35:33.5007174Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5007699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5008163Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5008627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2536
2022-12-01T10:35:33.5009071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2537
2022-12-01T10:35:33.5009670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5010102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5010674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5011141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5011701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5012142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5012705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5013170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5013590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5014059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5014768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5015246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5015910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5016600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5017529Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5018243Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5019071Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5019778Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5020102Z ok (5.378s)
2022-12-01T10:35:33.5020248Z 
2022-12-01T10:35:33.5020614Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5020940Z Ran 1 test in 5.378s
2022-12-01T10:35:33.5021099Z 
2022-12-01T10:35:33.5021253Z OK
2022-12-01T10:35:33.5021385Z 
2022-12-01T10:35:33.5021508Z Generating XML reports...
2022-12-01T10:35:33.5022049Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102420.xml
2022-12-01T10:35:33.5022718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5023168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5023724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5024191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5024420Z 
2022-12-01T10:35:33.5024528Z Running tests...
2022-12-01T10:35:33.5024933Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5025442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5025939Z   test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5026413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2647
2022-12-01T10:35:33.5026856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2648
2022-12-01T10:35:33.5027444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5027889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5028457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5028906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5029530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5029978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5030551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5030994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5031429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5031899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5032365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5032858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5033520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5034211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5035115Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5035834Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5036677Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5037430Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5038295Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5039041Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5039881Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5040585Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5041431Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5042114Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5042954Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5043653Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5044488Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5045180Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5045998Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5046701Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5047023Z ok (3.848s)
2022-12-01T10:35:33.5047170Z 
2022-12-01T10:35:33.5047441Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5047749Z Ran 1 test in 3.849s
2022-12-01T10:35:33.5047906Z 
2022-12-01T10:35:33.5047998Z OK
2022-12-01T10:35:33.5048130Z 
2022-12-01T10:35:33.5048253Z Generating XML reports...
2022-12-01T10:35:33.5048789Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102428.xml
2022-12-01T10:35:33.5049458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5049913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5050487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5050941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5051170Z 
2022-12-01T10:35:33.5051279Z Running tests...
2022-12-01T10:35:33.5051679Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5052186Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5052676Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5053151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2756
2022-12-01T10:35:33.5053594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2757
2022-12-01T10:35:33.5054233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5054929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5055610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5056061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5056638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5057079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5057645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5058087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5058526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5058999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5059464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5059962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5060619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5061303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5062204Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5062922Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5063761Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5064471Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5065308Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5065995Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5066839Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5067541Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5068385Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5069071Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5069908Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5070604Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5071504Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5072258Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5073070Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5073824Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5074149Z ok (5.267s)
2022-12-01T10:35:33.5074297Z 
2022-12-01T10:35:33.5074561Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5074871Z Ran 1 test in 5.267s
2022-12-01T10:35:33.5075034Z 
2022-12-01T10:35:33.5075126Z OK
2022-12-01T10:35:33.5075262Z 
2022-12-01T10:35:33.5075384Z Generating XML reports...
2022-12-01T10:35:33.5075921Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102434.xml
2022-12-01T10:35:33.5076603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5077048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5077619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5078069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5078298Z 
2022-12-01T10:35:33.5078407Z Running tests...
2022-12-01T10:35:33.5078813Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5079325Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5079817Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5080292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2867
2022-12-01T10:35:33.5080734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2868
2022-12-01T10:35:33.5081321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5081768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5082343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5082791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5083366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5083812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5084379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5084826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5085262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5085734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5086219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5086694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5087355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5088097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5089034Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5089787Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5090633Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5091338Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5092173Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5092867Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5093701Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5094624Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5094964Z ok (3.855s)
2022-12-01T10:35:33.5095111Z 
2022-12-01T10:35:33.5095369Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5095696Z Ran 1 test in 3.855s
2022-12-01T10:35:33.5095855Z 
2022-12-01T10:35:33.5095949Z OK
2022-12-01T10:35:33.5096079Z 
2022-12-01T10:35:33.5096207Z Generating XML reports...
2022-12-01T10:35:33.5096745Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102442.xml
2022-12-01T10:35:33.5097419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5097868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5098422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5098887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5099114Z 
2022-12-01T10:35:33.5099222Z Running tests...
2022-12-01T10:35:33.5099621Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5100128Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5100609Z   test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5101074Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2976
2022-12-01T10:35:33.5101502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2977
2022-12-01T10:35:33.5102106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5102550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5103126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5103577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5104152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5104595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5105268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5105753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5106254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5106724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5107192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5107683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5108336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5109022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5109931Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5110646Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5111482Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5112184Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5112489Z ok (3.865s)
2022-12-01T10:35:33.5112635Z 
2022-12-01T10:35:33.5112904Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5113228Z Ran 1 test in 3.865s
2022-12-01T10:35:33.5113387Z 
2022-12-01T10:35:33.5113464Z OK
2022-12-01T10:35:33.5113595Z 
2022-12-01T10:35:33.5113718Z Generating XML reports...
2022-12-01T10:35:33.5114267Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102448.xml
2022-12-01T10:35:33.5114945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5115376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5115944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5116411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5116639Z 
2022-12-01T10:35:33.5116746Z Running tests...
2022-12-01T10:35:33.5117129Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5117658Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5118137Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5118580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3085
2022-12-01T10:35:33.5119029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3086
2022-12-01T10:35:33.5119637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5120080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5120628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5121090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5121665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5122140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5122715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5123240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5123673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5124129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5124606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5125099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5125736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5126429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5127347Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5128066Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5128911Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1306: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2022-12-01T10:35:33.5129650Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2022-12-01T10:35:33.5129975Z ok (5.351s)
2022-12-01T10:35:33.5130120Z 
2022-12-01T10:35:33.5130392Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5130701Z Ran 1 test in 5.351s
2022-12-01T10:35:33.5130861Z 
2022-12-01T10:35:33.5130952Z OK
2022-12-01T10:35:33.5131082Z 
2022-12-01T10:35:33.5131211Z Generating XML reports...
2022-12-01T10:35:33.5131764Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102454.xml
2022-12-01T10:35:33.5132414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5132859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5133433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5133884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5134113Z 
2022-12-01T10:35:33.5134219Z Running tests...
2022-12-01T10:35:33.5134861Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5135401Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5135851Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5136463Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5136961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3196
2022-12-01T10:35:33.5137400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3197
2022-12-01T10:35:33.5137989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5138438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5139010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5139559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5140156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5140665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5141238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5141686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5142122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5142595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5143075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzt7vppn
2022-12-01T10:35:33.5143623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzt7vppn/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5144157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnhu_mgf
2022-12-01T10:35:33.5144686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnhu_mgf/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5145051Z ok (5.870s)
2022-12-01T10:35:33.5145197Z 
2022-12-01T10:35:33.5145470Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5145794Z Ran 1 test in 5.870s
2022-12-01T10:35:33.5145952Z 
2022-12-01T10:35:33.5146027Z OK
2022-12-01T10:35:33.5146160Z 
2022-12-01T10:35:33.5146284Z Generating XML reports...
2022-12-01T10:35:33.5146896Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102501.xml
2022-12-01T10:35:33.5147621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5148050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5148625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5149096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5149324Z 
2022-12-01T10:35:33.5149413Z Running tests...
2022-12-01T10:35:33.5149814Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5150341Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5150823Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5151325Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5151816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3311
2022-12-01T10:35:33.5152256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3312
2022-12-01T10:35:33.5152845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5153299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5153874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5154340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5154896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5155338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5155907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5156369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5156850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5157332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5157885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppr0ibzl
2022-12-01T10:35:33.5158409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppr0ibzl/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5158940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_wi3bvw
2022-12-01T10:35:33.5159470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_wi3bvw/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5159849Z ok (5.963s)
2022-12-01T10:35:33.5159977Z 
2022-12-01T10:35:33.5160251Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5160577Z Ran 1 test in 5.963s
2022-12-01T10:35:33.5160738Z 
2022-12-01T10:35:33.5160830Z OK
2022-12-01T10:35:33.5160962Z 
2022-12-01T10:35:33.5161071Z Generating XML reports...
2022-12-01T10:35:33.5161686Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102510.xml
2022-12-01T10:35:33.5162413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5162860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5163416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5163881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5164109Z 
2022-12-01T10:35:33.5164215Z Running tests...
2022-12-01T10:35:33.5164598Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5165134Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5165620Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5166121Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5166585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3426
2022-12-01T10:35:33.5167022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3427
2022-12-01T10:35:33.5167625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5168057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5168627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5169091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5169670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5170096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5170667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5171131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5171566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5172019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5172511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcuw31ugl
2022-12-01T10:35:33.5173046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcuw31ugl/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5173557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuwy9egy
2022-12-01T10:35:33.5174140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuwy9egy/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5174889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5175465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5175923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5176397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5177570Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5178290Z   warnings.warn(
2022-12-01T10:35:33.5179309Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5180033Z   warnings.warn(
2022-12-01T10:35:33.5180402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5180884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5181343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5181815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5182288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5182758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5183211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5183678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5184016Z ok (5.957s)
2022-12-01T10:35:33.5184160Z 
2022-12-01T10:35:33.5184416Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5184741Z Ran 1 test in 5.957s
2022-12-01T10:35:33.5184899Z 
2022-12-01T10:35:33.5184991Z OK
2022-12-01T10:35:33.5185121Z 
2022-12-01T10:35:33.5185244Z Generating XML reports...
2022-12-01T10:35:33.5185842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102518.xml
2022-12-01T10:35:33.5186570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5187019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5187575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5188049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5188280Z 
2022-12-01T10:35:33.5188387Z Running tests...
2022-12-01T10:35:33.5188791Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5189301Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5189782Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5190285Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5190740Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3541
2022-12-01T10:35:33.5191251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3542
2022-12-01T10:35:33.5191865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5192365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5192916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5193380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5193957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5194377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5194943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5195411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5195848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5196311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5196804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe14t9ixp
2022-12-01T10:35:33.5197344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe14t9ixp/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5197870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkvogxam
2022-12-01T10:35:33.5198387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkvogxam/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5198902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5199384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5199846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5200322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5201488Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5202208Z   warnings.warn(
2022-12-01T10:35:33.5203247Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5203944Z   warnings.warn(
2022-12-01T10:35:33.5204313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5204548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5204773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5205001Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5205223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5205429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5205665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5205948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5206057Z ok (5.958s)
2022-12-01T10:35:33.5206078Z 
2022-12-01T10:35:33.5206353Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5206516Z Ran 1 test in 5.958s
2022-12-01T10:35:33.5206536Z 
2022-12-01T10:35:33.5206628Z OK
2022-12-01T10:35:33.5206647Z 
2022-12-01T10:35:33.5206771Z Generating XML reports...
2022-12-01T10:35:33.5207218Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102526.xml
2022-12-01T10:35:33.5207593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5207773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5208154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5208349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5208371Z 
2022-12-01T10:35:33.5208480Z Running tests...
2022-12-01T10:35:33.5208745Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5209063Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5209327Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5209659Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5209878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3656
2022-12-01T10:35:33.5210087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3657
2022-12-01T10:35:33.5210459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5210637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5211018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5211214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5211580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5211755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5212117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5212309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5212536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5212771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5213031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd81wtkd
2022-12-01T10:35:33.5213303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd81wtkd/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5213563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7b3k_nf
2022-12-01T10:35:33.5213826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7b3k_nf/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5214045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5214279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5214717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5214951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5215054Z ok (5.851s)
2022-12-01T10:35:33.5215074Z 
2022-12-01T10:35:33.5215429Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5215553Z Ran 1 test in 5.851s
2022-12-01T10:35:33.5215629Z 
2022-12-01T10:35:33.5215724Z OK
2022-12-01T10:35:33.5215744Z 
2022-12-01T10:35:33.5215850Z Generating XML reports...
2022-12-01T10:35:33.5216321Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102534.xml
2022-12-01T10:35:33.5216692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5216867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5217245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5217438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5217458Z 
2022-12-01T10:35:33.5217566Z Running tests...
2022-12-01T10:35:33.5217834Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5218147Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5218395Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5218745Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5218962Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3771
2022-12-01T10:35:33.5219172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3772
2022-12-01T10:35:33.5219544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5219719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5220094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5220285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5220656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5220814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5221186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5221374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5221602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5221827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5222081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wc9z8vf
2022-12-01T10:35:33.5222354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wc9z8vf/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5222606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf6x3v4j
2022-12-01T10:35:33.5222856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf6x3v4j/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5223092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5223325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5223552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5223777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5223878Z ok (5.861s)
2022-12-01T10:35:33.5223898Z 
2022-12-01T10:35:33.5224166Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5224330Z Ran 1 test in 5.861s
2022-12-01T10:35:33.5224352Z 
2022-12-01T10:35:33.5224428Z OK
2022-12-01T10:35:33.5224466Z 
2022-12-01T10:35:33.5224573Z Generating XML reports...
2022-12-01T10:35:33.5225087Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102542.xml
2022-12-01T10:35:33.5225457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5225634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5226013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5226205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5226225Z 
2022-12-01T10:35:33.5226332Z Running tests...
2022-12-01T10:35:33.5226595Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5226891Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5227136Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5227516Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5227732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3886
2022-12-01T10:35:33.5227941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3887
2022-12-01T10:35:33.5228308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5228482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5228858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5229035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5229445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5229626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5230000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5258381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5258631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5258861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5259127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9nv5odp
2022-12-01T10:35:33.5259401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9nv5odp/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5259664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ajxt7x4
2022-12-01T10:35:33.5259933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ajxt7x4/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5260176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5260394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5261305Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:35:33.5262122Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:35:33.5262430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5262649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5262752Z ok (6.056s)
2022-12-01T10:35:33.5262774Z 
2022-12-01T10:35:33.5263088Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5263201Z Ran 1 test in 6.057s
2022-12-01T10:35:33.5263226Z 
2022-12-01T10:35:33.5263319Z OK
2022-12-01T10:35:33.5263339Z 
2022-12-01T10:35:33.5263461Z Generating XML reports...
2022-12-01T10:35:33.5263940Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102550.xml
2022-12-01T10:35:33.5264316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5264474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5264857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5265048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5265068Z 
2022-12-01T10:35:33.5265176Z Running tests...
2022-12-01T10:35:33.5265439Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5265750Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5265993Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5266371Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5266588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4001
2022-12-01T10:35:33.5266786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4002
2022-12-01T10:35:33.5267160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5267334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5267712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5267900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5268261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5268438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5268810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5268981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5269211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5269436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5269691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxdgp86v
2022-12-01T10:35:33.5269964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxdgp86v/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5270269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxg2g0q7
2022-12-01T10:35:33.5270546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxg2g0q7/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5270695Z ok (5.958s)
2022-12-01T10:35:33.5270716Z 
2022-12-01T10:35:33.5270987Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5271081Z Ran 1 test in 5.958s
2022-12-01T10:35:33.5271101Z 
2022-12-01T10:35:33.5271193Z OK
2022-12-01T10:35:33.5271212Z 
2022-12-01T10:35:33.5271334Z Generating XML reports...
2022-12-01T10:35:33.5271798Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102559.xml
2022-12-01T10:35:33.5272168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5272341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5272720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5272913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5272937Z 
2022-12-01T10:35:33.5273028Z Running tests...
2022-12-01T10:35:33.5273292Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5273601Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5273839Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5274110Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5274326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4116
2022-12-01T10:35:33.5274541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4117
2022-12-01T10:35:33.5274915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5275073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5275459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5275651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5276013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5276184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5276554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5276744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5276975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5277204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5277441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsb2olhvx
2022-12-01T10:35:33.5277713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsb2olhvx/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5277964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5961a5o
2022-12-01T10:35:33.5278227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5961a5o/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5278459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5278695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5278929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5279212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5279303Z ok (5.970s)
2022-12-01T10:35:33.5279340Z 
2022-12-01T10:35:33.5279641Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5279752Z Ran 1 test in 5.970s
2022-12-01T10:35:33.5279772Z 
2022-12-01T10:35:33.5279865Z OK
2022-12-01T10:35:33.5279884Z 
2022-12-01T10:35:33.5280007Z Generating XML reports...
2022-12-01T10:35:33.5280469Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102607.xml
2022-12-01T10:35:33.5280840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5281014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5281395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5281572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5281592Z 
2022-12-01T10:35:33.5281700Z Running tests...
2022-12-01T10:35:33.5281967Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5282275Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5282532Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5282803Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5283018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4231
2022-12-01T10:35:33.5283234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4232
2022-12-01T10:35:33.5283587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5283765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5284144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5284336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5284701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5284871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5285246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5285433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5285661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5285874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5286131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfn8_7_vm
2022-12-01T10:35:33.5286400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfn8_7_vm/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5286652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubow2w_e
2022-12-01T10:35:33.5286919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubow2w_e/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5287748Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:35:33.5288527Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:35:33.5289494Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5289609Z   warnings.warn(
2022-12-01T10:35:33.5290509Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5290622Z   warnings.warn(
2022-12-01T10:35:33.5290859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5291093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5291324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5291537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5291640Z ok (5.959s)
2022-12-01T10:35:33.5291661Z 
2022-12-01T10:35:33.5291928Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5292042Z Ran 1 test in 5.959s
2022-12-01T10:35:33.5292062Z 
2022-12-01T10:35:33.5292153Z OK
2022-12-01T10:35:33.5292173Z 
2022-12-01T10:35:33.5292292Z Generating XML reports...
2022-12-01T10:35:33.5292751Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102615.xml
2022-12-01T10:35:33.5293122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5293281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5293663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5293854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5293874Z 
2022-12-01T10:35:33.5293984Z Running tests...
2022-12-01T10:35:33.5294249Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5294818Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5295088Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5295362Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5295579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4346
2022-12-01T10:35:33.5295775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4347
2022-12-01T10:35:33.5296151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5296322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5296780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5296981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5297409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5297572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5297935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5298107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5298328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5298548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5298809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq8pprph
2022-12-01T10:35:33.5299082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq8pprph/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5299337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ojz73d7
2022-12-01T10:35:33.5299601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ojz73d7/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5300513Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5300626Z   warnings.warn(
2022-12-01T10:35:33.5301528Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5301636Z   warnings.warn(
2022-12-01T10:35:33.5301857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5302087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5302316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5302547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5302647Z ok (5.866s)
2022-12-01T10:35:33.5302668Z 
2022-12-01T10:35:33.5302930Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5303032Z Ran 1 test in 5.867s
2022-12-01T10:35:33.5303056Z 
2022-12-01T10:35:33.5303141Z OK
2022-12-01T10:35:33.5303160Z 
2022-12-01T10:35:33.5303266Z Generating XML reports...
2022-12-01T10:35:33.5303727Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102623.xml
2022-12-01T10:35:33.5304099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5304276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5304656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5304838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5304858Z 
2022-12-01T10:35:33.5304963Z Running tests...
2022-12-01T10:35:33.5305226Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5305586Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5305839Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5306165Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5306383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4461
2022-12-01T10:35:33.5306598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4462
2022-12-01T10:35:33.5306970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5307145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5307522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5307713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5308067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5308242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5308627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5308814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5309042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5309268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5309527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssdo98p2
2022-12-01T10:35:33.5309796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssdo98p2/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5310053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3y381l_z
2022-12-01T10:35:33.5310303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3y381l_z/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5310538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5310772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5311004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5311235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5311337Z ok (5.869s)
2022-12-01T10:35:33.5311357Z 
2022-12-01T10:35:33.5311627Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5311737Z Ran 1 test in 5.869s
2022-12-01T10:35:33.5311757Z 
2022-12-01T10:35:33.5311830Z OK
2022-12-01T10:35:33.5311849Z 
2022-12-01T10:35:33.5311971Z Generating XML reports...
2022-12-01T10:35:33.5312438Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102631.xml
2022-12-01T10:35:33.5312812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5312987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5313368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5313558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5313578Z 
2022-12-01T10:35:33.5313686Z Running tests...
2022-12-01T10:35:33.5313933Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5314245Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5314572Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5314820Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5315084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4576
2022-12-01T10:35:33.5315298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4577
2022-12-01T10:35:33.5315673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5315848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5316226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5316399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5316760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5316934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5317316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5317506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5317735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5317962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5318220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24usmq6i
2022-12-01T10:35:33.5318471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24usmq6i/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5318724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7v1cas8
2022-12-01T10:35:33.5318993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7v1cas8/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5319229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5319466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5319699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5319926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5320156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5320384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5320596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5320820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5320921Z ok (5.972s)
2022-12-01T10:35:33.5320944Z 
2022-12-01T10:35:33.5321216Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5321327Z Ran 1 test in 5.972s
2022-12-01T10:35:33.5321351Z 
2022-12-01T10:35:33.5321441Z OK
2022-12-01T10:35:33.5321461Z 
2022-12-01T10:35:33.5321582Z Generating XML reports...
2022-12-01T10:35:33.5322045Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102640.xml
2022-12-01T10:35:33.5322393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5322569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5322944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5323133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5323153Z 
2022-12-01T10:35:33.5323311Z Running tests...
2022-12-01T10:35:33.5323582Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5323894Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5324169Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5324433Z This unit test verifies whether the Future object is passed properly. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5324633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4691
2022-12-01T10:35:33.5324847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4692
2022-12-01T10:35:33.5325222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5325396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5325778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5325968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5326335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5326505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5326862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5327049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5327276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5327502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5327758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_dipxlj
2022-12-01T10:35:33.5328017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrxdemj0
2022-12-01T10:35:33.5328286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_dipxlj/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5328557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrxdemj0/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5328657Z ok (3.857s)
2022-12-01T10:35:33.5328678Z 
2022-12-01T10:35:33.5328928Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5329040Z Ran 1 test in 3.857s
2022-12-01T10:35:33.5329060Z 
2022-12-01T10:35:33.5329150Z OK
2022-12-01T10:35:33.5329170Z 
2022-12-01T10:35:33.5329292Z Generating XML reports...
2022-12-01T10:35:33.5329807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102648.xml
2022-12-01T10:35:33.5330182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5330357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5330734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5330910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5330947Z 
2022-12-01T10:35:33.5331036Z Running tests...
2022-12-01T10:35:33.5331301Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5331611Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5331841Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5332132Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5332400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4804
2022-12-01T10:35:33.5332625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4805
2022-12-01T10:35:33.5332999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5333204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5333585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5333774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5334145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5334525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5334919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5335111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5335342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5335555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5335809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgz1qey7
2022-12-01T10:35:33.5336078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgz1qey7/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5336328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb4dw7ue
2022-12-01T10:35:33.5336591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb4dw7ue/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5336692Z ok (5.471s)
2022-12-01T10:35:33.5336713Z 
2022-12-01T10:35:33.5336980Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5337096Z Ran 1 test in 5.471s
2022-12-01T10:35:33.5337117Z 
2022-12-01T10:35:33.5337209Z OK
2022-12-01T10:35:33.5337229Z 
2022-12-01T10:35:33.5337333Z Generating XML reports...
2022-12-01T10:35:33.5337804Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102654.xml
2022-12-01T10:35:33.5338170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5338343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5338718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5338906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5338926Z 
2022-12-01T10:35:33.5339034Z Running tests...
2022-12-01T10:35:33.5339296Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5339590Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5339809Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5340092Z DDP communication hook can only be registered once. This test validates whether ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5340308Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4919
2022-12-01T10:35:33.5340523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4920
2022-12-01T10:35:33.5340891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5341065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5341443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5341714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5342079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5342312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5342689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5342877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5343105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5343332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5343587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ppxxy9e
2022-12-01T10:35:33.5343841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz02mav3
2022-12-01T10:35:33.5344096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ppxxy9e/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5344364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz02mav3/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5344468Z ok (3.844s)
2022-12-01T10:35:33.5344489Z 
2022-12-01T10:35:33.5344758Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5344868Z Ran 1 test in 3.845s
2022-12-01T10:35:33.5344888Z 
2022-12-01T10:35:33.5344978Z OK
2022-12-01T10:35:33.5344997Z 
2022-12-01T10:35:33.5345119Z Generating XML reports...
2022-12-01T10:35:33.5345584Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102702.xml
2022-12-01T10:35:33.5345953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5346111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5346493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5346683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5346707Z 
2022-12-01T10:35:33.5346817Z Running tests...
2022-12-01T10:35:33.5347078Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5347385Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5347614Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5347897Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5348095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5028
2022-12-01T10:35:33.5348311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5029
2022-12-01T10:35:33.5348686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5348860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5349242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5349432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5349793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5349966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5350337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5350505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5350798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5351032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5351335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppue2y9ua
2022-12-01T10:35:33.5351592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpepu793g7
2022-12-01T10:35:33.5351862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppue2y9ua/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5352128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpepu793g7/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5352230Z ok (3.860s)
2022-12-01T10:35:33.5352251Z 
2022-12-01T10:35:33.5352504Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5352614Z Ran 1 test in 3.860s
2022-12-01T10:35:33.5352633Z 
2022-12-01T10:35:33.5352723Z OK
2022-12-01T10:35:33.5352742Z 
2022-12-01T10:35:33.5352865Z Generating XML reports...
2022-12-01T10:35:33.5353328Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102708.xml
2022-12-01T10:35:33.5353700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5353874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5354252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5354441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5354462Z 
2022-12-01T10:35:33.5354552Z Running tests...
2022-12-01T10:35:33.5354814Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5355122Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5355338Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5355609Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5355829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5171
2022-12-01T10:35:33.5356044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5172
2022-12-01T10:35:33.5356414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5356571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5356947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5357136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5357499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5357673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5358047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5358240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5358469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5358677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5358934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qi8bzhv
2022-12-01T10:35:33.5359206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qi8bzhv/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5359459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs3xiggf
2022-12-01T10:35:33.5359776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs3xiggf/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5359886Z ok (3.854s)
2022-12-01T10:35:33.5359905Z 
2022-12-01T10:35:33.5360175Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5360331Z Ran 1 test in 3.854s
2022-12-01T10:35:33.5360351Z 
2022-12-01T10:35:33.5360445Z OK
2022-12-01T10:35:33.5360464Z 
2022-12-01T10:35:33.5360568Z Generating XML reports...
2022-12-01T10:35:33.5361033Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102714.xml
2022-12-01T10:35:33.5361400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5361574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5361952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5362146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5362166Z 
2022-12-01T10:35:33.5362274Z Running tests...
2022-12-01T10:35:33.5362536Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5362833Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5363059Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5363339Z This test checks whether return annotation checked properly if defined. It also ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5363551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5280
2022-12-01T10:35:33.5363763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5281
2022-12-01T10:35:33.5364136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5364313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5364694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5364891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5365239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5365413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5365786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5365976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5366204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5366431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5366689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkh8unb9q
2022-12-01T10:35:33.5366941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuugk4rw9
2022-12-01T10:35:33.5367202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkh8unb9q/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5367465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuugk4rw9/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5367566Z ok (3.887s)
2022-12-01T10:35:33.5367586Z 
2022-12-01T10:35:33.5367853Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5367965Z Ran 1 test in 3.887s
2022-12-01T10:35:33.5367984Z 
2022-12-01T10:35:33.5368075Z OK
2022-12-01T10:35:33.5368094Z 
2022-12-01T10:35:33.5368216Z Generating XML reports...
2022-12-01T10:35:33.5368672Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102720.xml
2022-12-01T10:35:33.5369087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5369250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5369684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5369875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5369895Z 
2022-12-01T10:35:33.5370003Z Running tests...
2022-12-01T10:35:33.5370266Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5370574Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5370827Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5371099Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5371299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5393
2022-12-01T10:35:33.5371514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5394
2022-12-01T10:35:33.5371885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5372065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5372437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5372625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5372986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5373157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5373530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5373704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5373937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5374163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5374638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8rxyi3y
2022-12-01T10:35:33.5374902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu45xlti2
2022-12-01T10:35:33.5375173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8rxyi3y/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5375442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu45xlti2/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5376221Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:35:33.5376330Z ok (5.456s)
2022-12-01T10:35:33.5376351Z 
2022-12-01T10:35:33.5376626Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5376721Z Ran 1 test in 5.457s
2022-12-01T10:35:33.5376741Z 
2022-12-01T10:35:33.5376832Z OK
2022-12-01T10:35:33.5376851Z 
2022-12-01T10:35:33.5376971Z Generating XML reports...
2022-12-01T10:35:33.5377433Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102726.xml
2022-12-01T10:35:33.5377897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5378085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5378531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5378723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5378743Z 
2022-12-01T10:35:33.5378851Z Running tests...
2022-12-01T10:35:33.5379097Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5379408Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5379692Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5379907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5508
2022-12-01T10:35:33.5380124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5509
2022-12-01T10:35:33.5380491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5380668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5381043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5381214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5381576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5381747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5382117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5382307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5382539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5382770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5383029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96xsb2xf
2022-12-01T10:35:33.5383284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpja4a3qgk
2022-12-01T10:35:33.5383534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96xsb2xf/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5383798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpja4a3qgk/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5383899Z ok (5.378s)
2022-12-01T10:35:33.5383920Z 
2022-12-01T10:35:33.5384185Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5384295Z Ran 1 test in 5.378s
2022-12-01T10:35:33.5384318Z 
2022-12-01T10:35:33.5384409Z OK
2022-12-01T10:35:33.5384429Z 
2022-12-01T10:35:33.5384550Z Generating XML reports...
2022-12-01T10:35:33.5385008Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102734.xml
2022-12-01T10:35:33.5385363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5385537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5385917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5386107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5386127Z 
2022-12-01T10:35:33.5386232Z Running tests...
2022-12-01T10:35:33.5386493Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5386854Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5387170Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5387432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5623
2022-12-01T10:35:33.5387629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5624
2022-12-01T10:35:33.5388001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5388175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5388549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5388739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5389110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5389284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5389659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5389831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5390061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5390286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5390541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yhux7dn
2022-12-01T10:35:33.5390793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qc_blbc
2022-12-01T10:35:33.5391059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yhux7dn/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5391325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qc_blbc/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5391427Z ok (5.363s)
2022-12-01T10:35:33.5391451Z 
2022-12-01T10:35:33.5391717Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5391812Z Ran 1 test in 5.363s
2022-12-01T10:35:33.5391831Z 
2022-12-01T10:35:33.5391922Z OK
2022-12-01T10:35:33.5391941Z 
2022-12-01T10:35:33.5392062Z Generating XML reports...
2022-12-01T10:35:33.5392523Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102741.xml
2022-12-01T10:35:33.5392892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5393065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5393442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5393637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5393657Z 
2022-12-01T10:35:33.5393748Z Running tests...
2022-12-01T10:35:33.5394015Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5394324Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5394635Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5394852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5738
2022-12-01T10:35:33.5395064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5739
2022-12-01T10:35:33.5395430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5395603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5396030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5396212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5396625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5396800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5397169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5397355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5397583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5397810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5398071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu4c5wsg
2022-12-01T10:35:33.5398308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xl2k5xs
2022-12-01T10:35:33.5398580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu4c5wsg/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5398841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xl2k5xs/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5399751Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5399865Z   warnings.warn(
2022-12-01T10:35:33.5400770Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:35:33.5400884Z   warnings.warn(
2022-12-01T10:35:33.5400983Z ok (5.377s)
2022-12-01T10:35:33.5401004Z 
2022-12-01T10:35:33.5401267Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5401380Z Ran 1 test in 5.377s
2022-12-01T10:35:33.5401399Z 
2022-12-01T10:35:33.5401489Z OK
2022-12-01T10:35:33.5401508Z 
2022-12-01T10:35:33.5401613Z Generating XML reports...
2022-12-01T10:35:33.5402074Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102749.xml
2022-12-01T10:35:33.5402444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5402623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5403002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5403196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5403216Z 
2022-12-01T10:35:33.5403323Z Running tests...
2022-12-01T10:35:33.5403585Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5403877Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5404185Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5404401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5853
2022-12-01T10:35:33.5404617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5854
2022-12-01T10:35:33.5405040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5405221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5405649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5405838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5406200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5406356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5406730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5406917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5407150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5407372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5407633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjl6bnho
2022-12-01T10:35:33.5407900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjl6bnho/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5408151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjk65y5t
2022-12-01T10:35:33.5408399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjk65y5t/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5408638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5408871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5408971Z ok (5.847s)
2022-12-01T10:35:33.5408991Z 
2022-12-01T10:35:33.5409262Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5409373Z Ran 1 test in 5.848s
2022-12-01T10:35:33.5409393Z 
2022-12-01T10:35:33.5409488Z OK
2022-12-01T10:35:33.5409507Z 
2022-12-01T10:35:33.5409629Z Generating XML reports...
2022-12-01T10:35:33.5410092Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102756.xml
2022-12-01T10:35:33.5410443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5410617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5410993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5411182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5411202Z 
2022-12-01T10:35:33.5411308Z Running tests...
2022-12-01T10:35:33.5411573Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5411886Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5412205Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5412403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5968
2022-12-01T10:35:33.5412617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5969
2022-12-01T10:35:33.5412984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5413161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5413536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5413772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5414145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5414563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5414957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5415126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5415357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5415584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5415839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtf4k1vg
2022-12-01T10:35:33.5416107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtf4k1vg/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5416363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0iwek2kj
2022-12-01T10:35:33.5416630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0iwek2kj/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5416868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5417085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5417186Z ok (5.757s)
2022-12-01T10:35:33.5417207Z 
2022-12-01T10:35:33.5417477Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5417587Z Ran 1 test in 5.757s
2022-12-01T10:35:33.5417607Z 
2022-12-01T10:35:33.5417697Z OK
2022-12-01T10:35:33.5417717Z 
2022-12-01T10:35:33.5417838Z Generating XML reports...
2022-12-01T10:35:33.5418301Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102805.xml
2022-12-01T10:35:33.5418673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5418847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5419214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5419405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5419425Z 
2022-12-01T10:35:33.5419535Z Running tests...
2022-12-01T10:35:33.5419797Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5420106Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5420381Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5420598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6083
2022-12-01T10:35:33.5420815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6084
2022-12-01T10:35:33.5421169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5421346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5421722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5421912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5422272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5422443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5422812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5423081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5423320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5423589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5423742Z skip: Need at least 4 CUDA devices (3.872s)
2022-12-01T10:35:33.5423763Z 
2022-12-01T10:35:33.5424033Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5424143Z Ran 1 test in 3.872s
2022-12-01T10:35:33.5424163Z 
2022-12-01T10:35:33.5424272Z OK (skipped=1)
2022-12-01T10:35:33.5424291Z 
2022-12-01T10:35:33.5424414Z Generating XML reports...
2022-12-01T10:35:33.5424873Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102813.xml
2022-12-01T10:35:33.5425239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5425399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5425775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5425970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5425990Z 
2022-12-01T10:35:33.5426097Z Running tests...
2022-12-01T10:35:33.5426360Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5426670Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5426946Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5427162Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6186
2022-12-01T10:35:33.5427375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6187
2022-12-01T10:35:33.5427728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5427902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5428280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5428469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5428834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5429007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5429375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5429603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5429819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5430045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5430194Z skip: Need at least 8 CUDA devices (3.747s)
2022-12-01T10:35:33.5430218Z 
2022-12-01T10:35:33.5430485Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5430598Z Ran 1 test in 3.748s
2022-12-01T10:35:33.5430618Z 
2022-12-01T10:35:33.5430724Z OK (skipped=1)
2022-12-01T10:35:33.5430744Z 
2022-12-01T10:35:33.5430866Z Generating XML reports...
2022-12-01T10:35:33.5431322Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102819.xml
2022-12-01T10:35:33.5431689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5431845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5432272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5432468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5432529Z 
2022-12-01T10:35:33.5432639Z Running tests...
2022-12-01T10:35:33.5432904Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5433211Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5433484Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5433699Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6289
2022-12-01T10:35:33.5433896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6290
2022-12-01T10:35:33.5434263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5434439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5434816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5435010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5435373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5435543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5435919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5436105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5436320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5436548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5436809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpnvrtsb
2022-12-01T10:35:33.5437062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphm3knx3u
2022-12-01T10:35:33.5437336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpnvrtsb/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5437598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphm3knx3u/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5437833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5438064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5438148Z ok (3.875s)
2022-12-01T10:35:33.5438185Z 
2022-12-01T10:35:33.5438437Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5438546Z Ran 1 test in 3.875s
2022-12-01T10:35:33.5438566Z 
2022-12-01T10:35:33.5438656Z OK
2022-12-01T10:35:33.5438679Z 
2022-12-01T10:35:33.5438800Z Generating XML reports...
2022-12-01T10:35:33.5439258Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102825.xml
2022-12-01T10:35:33.5439632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5439807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5440182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5440356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5440376Z 
2022-12-01T10:35:33.5440482Z Running tests...
2022-12-01T10:35:33.5440744Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5441051Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5441405Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5441629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6402
2022-12-01T10:35:33.5441890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6403
2022-12-01T10:35:33.5442258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5442414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5442788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5442979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5443337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5443511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5443883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5444075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5444303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5444521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5444760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcdtvwfr
2022-12-01T10:35:33.5445014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwodsu71
2022-12-01T10:35:33.5445285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcdtvwfr/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5445551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwodsu71/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5445788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5446026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5446126Z ok (3.855s)
2022-12-01T10:35:33.5446146Z 
2022-12-01T10:35:33.5446415Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5446509Z Ran 1 test in 3.855s
2022-12-01T10:35:33.5446545Z 
2022-12-01T10:35:33.5446619Z OK
2022-12-01T10:35:33.5446638Z 
2022-12-01T10:35:33.5446760Z Generating XML reports...
2022-12-01T10:35:33.5447221Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102831.xml
2022-12-01T10:35:33.5447589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5447764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5448144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5448338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5448359Z 
2022-12-01T10:35:33.5448465Z Running tests...
2022-12-01T10:35:33.5448708Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5449019Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5449212Z   test_ignored_output (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5449467Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5449683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6515
2022-12-01T10:35:33.5449895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6516
2022-12-01T10:35:33.5450351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5450533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5450945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5451135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5451499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5451671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5452038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5452223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5452455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5452685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5452944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppf_32ekk
2022-12-01T10:35:33.5453185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx98qw5rn
2022-12-01T10:35:33.5453452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppf_32ekk/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5453711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx98qw5rn/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5453945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5454170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5454270Z ok (3.861s)
2022-12-01T10:35:33.5454291Z 
2022-12-01T10:35:33.5454808Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5454922Z Ran 1 test in 3.861s
2022-12-01T10:35:33.5454942Z 
2022-12-01T10:35:33.5455015Z OK
2022-12-01T10:35:33.5455038Z 
2022-12-01T10:35:33.5455156Z Generating XML reports...
2022-12-01T10:35:33.5455615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102837.xml
2022-12-01T10:35:33.5455984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5456154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5456530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5456719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5456739Z 
2022-12-01T10:35:33.5456843Z Running tests...
2022-12-01T10:35:33.5457108Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5457402Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5457636Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2022-12-01T10:35:33.5457892Z Test that the output of a model can be ignored and that there is no ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5458107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6658
2022-12-01T10:35:33.5458316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6659
2022-12-01T10:35:33.5458686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5458860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5459232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5459484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5459865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5460099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5460468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5460656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5460880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5461106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5461362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdtaz012
2022-12-01T10:35:33.5461603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60g3dq0o
2022-12-01T10:35:33.5461872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdtaz012/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5462137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60g3dq0o/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5462238Z ok (3.879s)
2022-12-01T10:35:33.5462258Z 
2022-12-01T10:35:33.5462526Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5462633Z Ran 1 test in 3.880s
2022-12-01T10:35:33.5462653Z 
2022-12-01T10:35:33.5462743Z OK
2022-12-01T10:35:33.5462763Z 
2022-12-01T10:35:33.5462885Z Generating XML reports...
2022-12-01T10:35:33.5463346Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102843.xml
2022-12-01T10:35:33.5463702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5463879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5464253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5464444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5464464Z 
2022-12-01T10:35:33.5464571Z Running tests...
2022-12-01T10:35:33.5464835Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5465142Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5465414Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5465612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6801
2022-12-01T10:35:33.5465825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6802
2022-12-01T10:35:33.5466194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5466367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5466740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5466927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5467292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5467463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5467830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5467998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5468223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5468820Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5469411Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5469944Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5470484Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5471022Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5471559Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5471784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5472308Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5472840Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5473365Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5473901Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5474474Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5475011Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:35:33.5475178Z ok (3.720s)
2022-12-01T10:35:33.5475198Z 
2022-12-01T10:35:33.5475467Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5475576Z Ran 1 test in 3.720s
2022-12-01T10:35:33.5475597Z 
2022-12-01T10:35:33.5475681Z OK
2022-12-01T10:35:33.5475700Z 
2022-12-01T10:35:33.5475817Z Generating XML reports...
2022-12-01T10:35:33.5476271Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102849.xml
2022-12-01T10:35:33.5476630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5476799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5477174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5477361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5477382Z 
2022-12-01T10:35:33.5477485Z Running tests...
2022-12-01T10:35:33.5477738Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5478042Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5478306Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5478507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6904
2022-12-01T10:35:33.5478715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6905
2022-12-01T10:35:33.5479076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5479241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5479609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5479789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5480141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5480301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5480660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5480831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5481053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5481262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5481500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5481735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5482126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5482511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:35:33.5482754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14_mqt10
2022-12-01T10:35:33.5483065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14_mqt10/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5483311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2epu46k
2022-12-01T10:35:33.5483614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2epu46k/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5483838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5484059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5484282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5484503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5484597Z ok (5.976s)
2022-12-01T10:35:33.5484618Z 
2022-12-01T10:35:33.5484884Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5484977Z Ran 1 test in 5.976s
2022-12-01T10:35:33.5484997Z 
2022-12-01T10:35:33.5485083Z OK
2022-12-01T10:35:33.5485103Z 
2022-12-01T10:35:33.5485218Z Generating XML reports...
2022-12-01T10:35:33.5485676Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102855.xml
2022-12-01T10:35:33.5486042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5486206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5486579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5486761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5486781Z 
2022-12-01T10:35:33.5486878Z Running tests...
2022-12-01T10:35:33.5487122Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5487426Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5487684Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5487897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7019
2022-12-01T10:35:33.5488100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7020
2022-12-01T10:35:33.5488458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5488624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5488994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5489167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5489520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5489687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5490049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5490229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5490449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5490667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5490911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdif_myx
2022-12-01T10:35:33.5491148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnki5f5b
2022-12-01T10:35:33.5491409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdif_myx/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5491722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnki5f5b/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5491821Z ok (3.851s)
2022-12-01T10:35:33.5491841Z 
2022-12-01T10:35:33.5492098Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5492249Z Ran 1 test in 3.852s
2022-12-01T10:35:33.5492270Z 
2022-12-01T10:35:33.5492359Z OK
2022-12-01T10:35:33.5492378Z 
2022-12-01T10:35:33.5492492Z Generating XML reports...
2022-12-01T10:35:33.5492951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102903.xml
2022-12-01T10:35:33.5493307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5493472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5493843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5494030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5494051Z 
2022-12-01T10:35:33.5494150Z Running tests...
2022-12-01T10:35:33.5494623Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5494958Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5495237Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5495435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7162
2022-12-01T10:35:33.5495645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7163
2022-12-01T10:35:33.5496005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5496173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5496544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5496730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5497090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5497255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5497617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5497788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5498012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5498226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5498476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur3khl_x
2022-12-01T10:35:33.5498728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3ubf60q
2022-12-01T10:35:33.5498991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur3khl_x/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5499252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3ubf60q/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5499347Z ok (3.866s)
2022-12-01T10:35:33.5499367Z 
2022-12-01T10:35:33.5499617Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5499723Z Ran 1 test in 3.866s
2022-12-01T10:35:33.5499743Z 
2022-12-01T10:35:33.5499827Z OK
2022-12-01T10:35:33.5499847Z 
2022-12-01T10:35:33.5499966Z Generating XML reports...
2022-12-01T10:35:33.5500423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102909.xml
2022-12-01T10:35:33.5500786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5501047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5501437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5501687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5501707Z 
2022-12-01T10:35:33.5501798Z Running tests...
2022-12-01T10:35:33.5502067Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5502373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5502640Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5502846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7305
2022-12-01T10:35:33.5503049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7306
2022-12-01T10:35:33.5503412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5503579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5503941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5504128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5504485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5504649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5505012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5505194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5505422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5505641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5505898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz4hx1ly
2022-12-01T10:35:33.5506151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz4hx1ly/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5506396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv15fw6vx
2022-12-01T10:35:33.5506654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv15fw6vx/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5506885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5507113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5507210Z ok (6.870s)
2022-12-01T10:35:33.5507231Z 
2022-12-01T10:35:33.5507495Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5507601Z Ran 1 test in 6.870s
2022-12-01T10:35:33.5507621Z 
2022-12-01T10:35:33.5507694Z OK
2022-12-01T10:35:33.5507728Z 
2022-12-01T10:35:33.5507834Z Generating XML reports...
2022-12-01T10:35:33.5508295Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102915.xml
2022-12-01T10:35:33.5508657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5508826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5509195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5509382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5509402Z 
2022-12-01T10:35:33.5509505Z Running tests...
2022-12-01T10:35:33.5509818Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5510126Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5510465Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5510674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7420
2022-12-01T10:35:33.5510884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7421
2022-12-01T10:35:33.5511252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5511418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5511783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5511961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5512312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5512474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5512837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5513012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5513236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5513456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5513705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp877g26c5
2022-12-01T10:35:33.5513964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp877g26c5/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5514218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy54b7cht
2022-12-01T10:35:33.5514468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy54b7cht/_remote_module_non_scriptable.py
2022-12-01T10:35:33.5514702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5514921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:35:33.5515015Z ok (6.170s)
2022-12-01T10:35:33.5515035Z 
2022-12-01T10:35:33.5515301Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5515406Z Ran 1 test in 6.170s
2022-12-01T10:35:33.5515426Z 
2022-12-01T10:35:33.5515512Z OK
2022-12-01T10:35:33.5515532Z 
2022-12-01T10:35:33.5515649Z Generating XML reports...
2022-12-01T10:35:33.5516092Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102925.xml
2022-12-01T10:35:33.5516459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5516629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5517007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5517191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5517210Z 
2022-12-01T10:35:33.5517307Z Running tests...
2022-12-01T10:35:33.5517566Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5517868Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5518186Z   test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5518385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7535
2022-12-01T10:35:33.5518798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5518969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5519387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5519570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5519793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5520032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5520428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:35:33.5520512Z ok (3.655s)
2022-12-01T10:35:33.5520544Z 
2022-12-01T10:35:33.5520786Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5520893Z Ran 1 test in 3.655s
2022-12-01T10:35:33.5520913Z 
2022-12-01T10:35:33.5521003Z OK
2022-12-01T10:35:33.5521022Z 
2022-12-01T10:35:33.5521138Z Generating XML reports...
2022-12-01T10:35:33.5521691Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221201102933.xml
2022-12-01T10:35:33.5522054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5522221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5522593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5522766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5522796Z 
2022-12-01T10:35:33.5522886Z Running tests...
2022-12-01T10:35:33.5523139Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5523448Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5523694Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5523906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7607
2022-12-01T10:35:33.5524114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7608
2022-12-01T10:35:33.5524320Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7609
2022-12-01T10:35:33.5524511Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7610
2022-12-01T10:35:33.5524876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5525044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5525418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5525600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5525960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5526126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5526498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5526681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5527020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5527183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5527544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5527770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5528144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5528357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5528723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5528903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5529115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5529337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5529595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5529820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5529916Z ok (3.954s)
2022-12-01T10:35:33.5529936Z 
2022-12-01T10:35:33.5530195Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5530303Z Ran 1 test in 3.955s
2022-12-01T10:35:33.5530322Z 
2022-12-01T10:35:33.5530409Z OK
2022-12-01T10:35:33.5530428Z 
2022-12-01T10:35:33.5530546Z Generating XML reports...
2022-12-01T10:35:33.5530960Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102939.xml
2022-12-01T10:35:33.5531320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5531491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5531862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5532049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5532069Z 
2022-12-01T10:35:33.5532170Z Running tests...
2022-12-01T10:35:33.5532428Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5532738Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5532977Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5533188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7790
2022-12-01T10:35:33.5533395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7791
2022-12-01T10:35:33.5533601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7792
2022-12-01T10:35:33.5533800Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7793
2022-12-01T10:35:33.5534163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5534526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5534923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5535112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5535460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5535632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5535998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5536181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5536533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5536774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5537156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5538231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5538592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5538746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5539108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5539287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5539505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5539724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5539945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5540161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5540262Z ok (5.774s)
2022-12-01T10:35:33.5540282Z 
2022-12-01T10:35:33.5540530Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5540633Z Ran 1 test in 5.774s
2022-12-01T10:35:33.5540653Z 
2022-12-01T10:35:33.5540737Z OK
2022-12-01T10:35:33.5540757Z 
2022-12-01T10:35:33.5540879Z Generating XML reports...
2022-12-01T10:35:33.5541363Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102945.xml
2022-12-01T10:35:33.5541721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5541884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5542250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5542423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5542455Z 
2022-12-01T10:35:33.5542545Z Running tests...
2022-12-01T10:35:33.5542798Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5543096Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5543336Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5543540Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7977
2022-12-01T10:35:33.5543742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7978
2022-12-01T10:35:33.5543946Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7979
2022-12-01T10:35:33.5544147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7980
2022-12-01T10:35:33.5544499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5544669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5545028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5545200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5545577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5545764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5546139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5546382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5546736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5546959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5547329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5547516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5547876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5548047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5548413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5548597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5548829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5549038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5549261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5549484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5549585Z ok (3.947s)
2022-12-01T10:35:33.5549605Z 
2022-12-01T10:35:33.5549868Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5549977Z Ran 1 test in 3.947s
2022-12-01T10:35:33.5549997Z 
2022-12-01T10:35:33.5550087Z OK
2022-12-01T10:35:33.5550107Z 
2022-12-01T10:35:33.5550228Z Generating XML reports...
2022-12-01T10:35:33.5550643Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102953.xml
2022-12-01T10:35:33.5551011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5551183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5551559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5551746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5551766Z 
2022-12-01T10:35:33.5551872Z Running tests...
2022-12-01T10:35:33.5552134Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5552441Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5552704Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5552905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8160
2022-12-01T10:35:33.5553124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8161
2022-12-01T10:35:33.5553335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8162
2022-12-01T10:35:33.5553546Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8163
2022-12-01T10:35:33.5553916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5554088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5554462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5554651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5554996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5555164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5555588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5555779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5556197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5556367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5556735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5556919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5557279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5557433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5557800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5557987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5558217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5558444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5558667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5558884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5559127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5559351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5559592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:35:33.5559833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:35:33.5560235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5560634Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5561019Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5561406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5562160Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5562275Z   warnings.warn(
2022-12-01T10:35:33.5563006Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5563104Z   warnings.warn(
2022-12-01T10:35:33.5563835Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5563943Z   warnings.warn(
2022-12-01T10:35:33.5564732Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5564846Z   warnings.warn(
2022-12-01T10:35:33.5564990Z ok (3.951s)
2022-12-01T10:35:33.5565010Z 
2022-12-01T10:35:33.5565279Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5565390Z Ran 1 test in 3.951s
2022-12-01T10:35:33.5565410Z 
2022-12-01T10:35:33.5565500Z OK
2022-12-01T10:35:33.5565519Z 
2022-12-01T10:35:33.5565641Z Generating XML reports...
2022-12-01T10:35:33.5566057Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102959.xml
2022-12-01T10:35:33.5566424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5566597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5566979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5567170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5567191Z 
2022-12-01T10:35:33.5567300Z Running tests...
2022-12-01T10:35:33.5567565Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5567874Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5568124Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5568341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8343
2022-12-01T10:35:33.5568553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8344
2022-12-01T10:35:33.5568765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8345
2022-12-01T10:35:33.5568977Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8346
2022-12-01T10:35:33.5569350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5569525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5569900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5570073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5570433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5570622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5570997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5571184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5571548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5571722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5572103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5572291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5572631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5572802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5573174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5573360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5573590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5573877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5574114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5574610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5575370Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5575465Z   warnings.warn(
2022-12-01T10:35:33.5576200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5576311Z   warnings.warn(
2022-12-01T10:35:33.5577033Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5577147Z   warnings.warn(
2022-12-01T10:35:33.5577863Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5577971Z   warnings.warn(
2022-12-01T10:35:33.5578070Z ok (3.956s)
2022-12-01T10:35:33.5578091Z 
2022-12-01T10:35:33.5578357Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5578467Z Ran 1 test in 3.956s
2022-12-01T10:35:33.5578491Z 
2022-12-01T10:35:33.5578565Z OK
2022-12-01T10:35:33.5578584Z 
2022-12-01T10:35:33.5578705Z Generating XML reports...
2022-12-01T10:35:33.5579137Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103005.xml
2022-12-01T10:35:33.5579508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5579683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5580059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5580250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5580270Z 
2022-12-01T10:35:33.5580377Z Running tests...
2022-12-01T10:35:33.5580622Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5580936Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5581206Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5581426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8526
2022-12-01T10:35:33.5581640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8527
2022-12-01T10:35:33.5581850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8528
2022-12-01T10:35:33.5582056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8529
2022-12-01T10:35:33.5582427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5582601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5582960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5583234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5583618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5583894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5584267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5584454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5584812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5584982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5585332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5585521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5585886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5586062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5586428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5586612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5586838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5587068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5587289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5587497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5587601Z ok (4.058s)
2022-12-01T10:35:33.5587623Z 
2022-12-01T10:35:33.5587889Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5588003Z Ran 1 test in 4.058s
2022-12-01T10:35:33.5588023Z 
2022-12-01T10:35:33.5588114Z OK
2022-12-01T10:35:33.5588133Z 
2022-12-01T10:35:33.5588256Z Generating XML reports...
2022-12-01T10:35:33.5588687Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103012.xml
2022-12-01T10:35:33.5589050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5589205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5589581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5589768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5589788Z 
2022-12-01T10:35:33.5589902Z Running tests...
2022-12-01T10:35:33.5590165Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5590474Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5590726Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5590941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8709
2022-12-01T10:35:33.5591156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8710
2022-12-01T10:35:33.5591352Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8711
2022-12-01T10:35:33.5591562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8712
2022-12-01T10:35:33.5591930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5592151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5592542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5592778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5593140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5593310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5593665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5593851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5594207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5594377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5594748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5594933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5595298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5595469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5595841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5596009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5596239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5596465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5596691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5596919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5597023Z ok (4.452s)
2022-12-01T10:35:33.5597044Z 
2022-12-01T10:35:33.5597309Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5597419Z Ran 1 test in 4.452s
2022-12-01T10:35:33.5597438Z 
2022-12-01T10:35:33.5597512Z OK
2022-12-01T10:35:33.5597531Z 
2022-12-01T10:35:33.5597652Z Generating XML reports...
2022-12-01T10:35:33.5598088Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103018.xml
2022-12-01T10:35:33.5598454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5598627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5599001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5599190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5599214Z 
2022-12-01T10:35:33.5599322Z Running tests...
2022-12-01T10:35:33.5599585Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5599877Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5600135Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5600350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8916
2022-12-01T10:35:33.5600564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8917
2022-12-01T10:35:33.5600776Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8918
2022-12-01T10:35:33.5600987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8919
2022-12-01T10:35:33.5601401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5601621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5601980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5602167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5602526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5602697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5603065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5603250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5603610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5603781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5604147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5604333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5604685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5604854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5605223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5605411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5605642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5605869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5606093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5606304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5606406Z ok (7.264s)
2022-12-01T10:35:33.5606426Z 
2022-12-01T10:35:33.5606689Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5606800Z Ran 1 test in 7.265s
2022-12-01T10:35:33.5606820Z 
2022-12-01T10:35:33.5606911Z OK
2022-12-01T10:35:33.5606931Z 
2022-12-01T10:35:33.5607051Z Generating XML reports...
2022-12-01T10:35:33.5607483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103025.xml
2022-12-01T10:35:33.5607855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5608011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5608384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5608575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5608595Z 
2022-12-01T10:35:33.5608703Z Running tests...
2022-12-01T10:35:33.5608963Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5609272Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5609523Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5609739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9127
2022-12-01T10:35:33.5609951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9128
2022-12-01T10:35:33.5610198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9129
2022-12-01T10:35:33.5610419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9130
2022-12-01T10:35:33.5610839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5611013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5611387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5611576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5611935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5612105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5612464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5612651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5613015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5613185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5613555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5613742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5614100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5614269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5614865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5615043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5615273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5615495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5615722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5615946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5616044Z ok (3.966s)
2022-12-01T10:35:33.5616065Z 
2022-12-01T10:35:33.5616333Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5616444Z Ran 1 test in 3.966s
2022-12-01T10:35:33.5616463Z 
2022-12-01T10:35:33.5616537Z OK
2022-12-01T10:35:33.5616556Z 
2022-12-01T10:35:33.5616676Z Generating XML reports...
2022-12-01T10:35:33.5617115Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103034.xml
2022-12-01T10:35:33.5617479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5617656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5618032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5618219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5618239Z 
2022-12-01T10:35:33.5618347Z Running tests...
2022-12-01T10:35:33.5618611Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5618905Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5619162Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5619457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9310
2022-12-01T10:35:33.5619684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9311
2022-12-01T10:35:33.5619953Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9312
2022-12-01T10:35:33.5620161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9313
2022-12-01T10:35:33.5620532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5620708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5621054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5621228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5621601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5621796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5622169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5622358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5622718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5622889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5623239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5623424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5623788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5623963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5624329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5624515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5624744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5624968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5625187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5625395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5625495Z ok (5.774s)
2022-12-01T10:35:33.5625515Z 
2022-12-01T10:35:33.5625777Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5625888Z Ran 1 test in 5.774s
2022-12-01T10:35:33.5625908Z 
2022-12-01T10:35:33.5626001Z OK
2022-12-01T10:35:33.5626021Z 
2022-12-01T10:35:33.5626143Z Generating XML reports...
2022-12-01T10:35:33.5626572Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103040.xml
2022-12-01T10:35:33.5626940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5627096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5627469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5627655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5627675Z 
2022-12-01T10:35:33.5627783Z Running tests...
2022-12-01T10:35:33.5628042Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5628409Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5628695Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5628959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9497
2022-12-01T10:35:33.5629174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9498
2022-12-01T10:35:33.5629369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9499
2022-12-01T10:35:33.5629616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9500
2022-12-01T10:35:33.5629988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5630159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5630529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5630720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5631079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5631254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5631606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5631790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5632156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5632327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5632695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5632883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5633241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5633417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5633780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5633947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5634175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5634400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5634625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5634843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5634946Z ok (5.856s)
2022-12-01T10:35:33.5634967Z 
2022-12-01T10:35:33.5635229Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5635343Z Ran 1 test in 5.856s
2022-12-01T10:35:33.5635363Z 
2022-12-01T10:35:33.5635437Z OK
2022-12-01T10:35:33.5635456Z 
2022-12-01T10:35:33.5635579Z Generating XML reports...
2022-12-01T10:35:33.5636011Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103049.xml
2022-12-01T10:35:33.5636380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5636553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5636929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5637118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5637138Z 
2022-12-01T10:35:33.5637295Z Running tests...
2022-12-01T10:35:33.5637566Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5637859Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5638181Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5638397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9684
2022-12-01T10:35:33.5638609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9685
2022-12-01T10:35:33.5638820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9686
2022-12-01T10:35:33.5639030Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9687
2022-12-01T10:35:33.5639399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5639576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5639936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5640127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5640492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5640663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5641028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5641213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5641572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5641745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5642120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5642296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5642654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5642824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5643195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5643380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5643606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5643834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5644062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5644272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5644377Z ok (4.066s)
2022-12-01T10:35:33.5644397Z 
2022-12-01T10:35:33.5644663Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5644774Z Ran 1 test in 4.066s
2022-12-01T10:35:33.5644794Z 
2022-12-01T10:35:33.5644883Z OK
2022-12-01T10:35:33.5644902Z 
2022-12-01T10:35:33.5645023Z Generating XML reports...
2022-12-01T10:35:33.5645453Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103057.xml
2022-12-01T10:35:33.5645820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5645976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5646401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5646599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5646660Z 
2022-12-01T10:35:33.5646772Z Running tests...
2022-12-01T10:35:33.5647040Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5647349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5647598Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5647815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9867
2022-12-01T10:35:33.5648027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9868
2022-12-01T10:35:33.5648222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9869
2022-12-01T10:35:33.5648433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9870
2022-12-01T10:35:33.5648807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5648985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5649360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5649548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5649908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5650080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5650434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5650624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5650983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5651155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5651533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5651720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5652078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5652248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5652619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5652789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5653021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5653248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5653475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5653699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5653799Z ok (3.958s)
2022-12-01T10:35:33.5653819Z 
2022-12-01T10:35:33.5654081Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5654194Z Ran 1 test in 3.959s
2022-12-01T10:35:33.5654214Z 
2022-12-01T10:35:33.5654288Z OK
2022-12-01T10:35:33.5654307Z 
2022-12-01T10:35:33.5654642Z Generating XML reports...
2022-12-01T10:35:33.5655081Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103103.xml
2022-12-01T10:35:33.5655524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5655711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5656093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5656348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5656368Z 
2022-12-01T10:35:33.5656475Z Running tests...
2022-12-01T10:35:33.5656736Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5657031Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5657297Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5657514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10050
2022-12-01T10:35:33.5657727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10051
2022-12-01T10:35:33.5657945Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10052
2022-12-01T10:35:33.5658156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10053
2022-12-01T10:35:33.5658532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5658706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5659068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5659257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5659617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5659785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5660155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5660344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5660707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5660877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5661243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5661411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5661778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5661946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5662313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5662503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5662733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5662966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5663184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5663392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5663635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.5663878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:35:33.5664120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:35:33.5664407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:35:33.5664823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5665268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5665656Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5666047Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:35:33.5666787Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5666887Z   warnings.warn(
2022-12-01T10:35:33.5667616Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5667731Z   warnings.warn(
2022-12-01T10:35:33.5668463Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5668570Z   warnings.warn(
2022-12-01T10:35:33.5669300Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:35:33.5669409Z   warnings.warn(
2022-12-01T10:35:33.5669509Z ok (3.953s)
2022-12-01T10:35:33.5669533Z 
2022-12-01T10:35:33.5669799Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5669909Z Ran 1 test in 3.953s
2022-12-01T10:35:33.5669929Z 
2022-12-01T10:35:33.5670002Z OK
2022-12-01T10:35:33.5670022Z 
2022-12-01T10:35:33.5670144Z Generating XML reports...
2022-12-01T10:35:33.5670577Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103109.xml
2022-12-01T10:35:33.5670945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5671118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5671495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5671688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5671708Z 
2022-12-01T10:35:33.5671816Z Running tests...
2022-12-01T10:35:33.5672067Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5672376Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5672641Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5672859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10233
2022-12-01T10:35:33.5673074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10234
2022-12-01T10:35:33.5673289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10235
2022-12-01T10:35:33.5673501Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10236
2022-12-01T10:35:33.5673925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5674108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5674521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5674711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5675072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5675245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5675615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5675799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5676161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5676332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5676694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5676882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5677243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5677416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5677784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5677969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5678199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5678432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5678657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5678870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5678971Z ok (3.936s)
2022-12-01T10:35:33.5678991Z 
2022-12-01T10:35:33.5679257Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5679369Z Ran 1 test in 3.936s
2022-12-01T10:35:33.5679389Z 
2022-12-01T10:35:33.5679479Z OK
2022-12-01T10:35:33.5679499Z 
2022-12-01T10:35:33.5679621Z Generating XML reports...
2022-12-01T10:35:33.5680056Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103115.xml
2022-12-01T10:35:33.5680421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5680579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5680956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5681148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5681167Z 
2022-12-01T10:35:33.5681275Z Running tests...
2022-12-01T10:35:33.5681537Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5681847Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5682113Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5682329Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10416
2022-12-01T10:35:33.5682528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10417
2022-12-01T10:35:33.5682792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10418
2022-12-01T10:35:33.5683011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10419
2022-12-01T10:35:33.5683431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5683605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5683982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5684171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5684535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5684706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5685052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5685229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5685604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5685795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5686167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5686352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5686714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5686888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5687239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5687429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5687658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5687889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5688108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5688331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5688432Z ok (3.909s)
2022-12-01T10:35:33.5688453Z 
2022-12-01T10:35:33.5688721Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5688830Z Ran 1 test in 3.909s
2022-12-01T10:35:33.5688851Z 
2022-12-01T10:35:33.5688924Z OK
2022-12-01T10:35:33.5688944Z 
2022-12-01T10:35:33.5689068Z Generating XML reports...
2022-12-01T10:35:33.5689505Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103122.xml
2022-12-01T10:35:33.5689870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5690047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5690423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5690612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5690633Z 
2022-12-01T10:35:33.5690741Z Running tests...
2022-12-01T10:35:33.5690988Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5691298Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5691573Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5691852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10599
2022-12-01T10:35:33.5692076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10600
2022-12-01T10:35:33.5692288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10601
2022-12-01T10:35:33.5692542Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10602
2022-12-01T10:35:33.5692915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5693088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5693447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5693636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5693996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5694171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5694758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5694959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5695328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5695498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5695847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5696035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5696395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5696571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5696937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5697124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5697351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5697579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5697802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5698006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5698107Z ok (5.657s)
2022-12-01T10:35:33.5698127Z 
2022-12-01T10:35:33.5698392Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5698502Z Ran 1 test in 5.658s
2022-12-01T10:35:33.5698521Z 
2022-12-01T10:35:33.5698611Z OK
2022-12-01T10:35:33.5698634Z 
2022-12-01T10:35:33.5698756Z Generating XML reports...
2022-12-01T10:35:33.5699190Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103128.xml
2022-12-01T10:35:33.5699562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5699718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5700089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5700275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5700296Z 
2022-12-01T10:35:33.5700405Z Running tests...
2022-12-01T10:35:33.5700667Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5700977Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5701321Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5701550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10786
2022-12-01T10:35:33.5701811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10787
2022-12-01T10:35:33.5702025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10788
2022-12-01T10:35:33.5702238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10789
2022-12-01T10:35:33.5702614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5702789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5703164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5703356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5703716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5703891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5704243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5704430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5704783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5704953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5705324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5705509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5705869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5706040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5706398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5706587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5706814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5707038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5707261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5707484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5707584Z ok (4.356s)
2022-12-01T10:35:33.5707604Z 
2022-12-01T10:35:33.5707876Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5707988Z Ran 1 test in 4.356s
2022-12-01T10:35:33.5708011Z 
2022-12-01T10:35:33.5708085Z OK
2022-12-01T10:35:33.5708105Z 
2022-12-01T10:35:33.5708226Z Generating XML reports...
2022-12-01T10:35:33.5708660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103136.xml
2022-12-01T10:35:33.5709026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5709198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5709573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5709762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5709781Z 
2022-12-01T10:35:33.5709888Z Running tests...
2022-12-01T10:35:33.5710180Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5710505Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5710808Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5711024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10993
2022-12-01T10:35:33.5711239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10994
2022-12-01T10:35:33.5711451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10995
2022-12-01T10:35:33.5711662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10996
2022-12-01T10:35:33.5712032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5712205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5712569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5712764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5713125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5713295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5713664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5713853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5714208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5714380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5714742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5714931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5715291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5715461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5715832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5716015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5716244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5716469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5716692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5716901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5717002Z ok (4.157s)
2022-12-01T10:35:33.5717026Z 
2022-12-01T10:35:33.5717289Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5717399Z Ran 1 test in 4.157s
2022-12-01T10:35:33.5717419Z 
2022-12-01T10:35:33.5717509Z OK
2022-12-01T10:35:33.5717528Z 
2022-12-01T10:35:33.5717650Z Generating XML reports...
2022-12-01T10:35:33.5718080Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103142.xml
2022-12-01T10:35:33.5718443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5718598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5719020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5719216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5719236Z 
2022-12-01T10:35:33.5719389Z Running tests...
2022-12-01T10:35:33.5719652Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5719960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5720218Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5720434Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11200
2022-12-01T10:35:33.5720634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11201
2022-12-01T10:35:33.5720851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11202
2022-12-01T10:35:33.5721062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11203
2022-12-01T10:35:33.5721430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5721605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5721984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5722174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5722540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5722711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5723066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5723252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5723616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5723788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5724159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5724345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5724703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5724876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5725227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5725413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5725639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5725867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5726090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5726312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5726411Z ok (6.125s)
2022-12-01T10:35:33.5726432Z 
2022-12-01T10:35:33.5726696Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5726805Z Ran 1 test in 6.125s
2022-12-01T10:35:33.5726825Z 
2022-12-01T10:35:33.5726899Z OK
2022-12-01T10:35:33.5726918Z 
2022-12-01T10:35:33.5727039Z Generating XML reports...
2022-12-01T10:35:33.5727468Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103149.xml
2022-12-01T10:35:33.5727833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5728101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5728490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5728726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5728746Z 
2022-12-01T10:35:33.5728853Z Running tests...
2022-12-01T10:35:33.5729100Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5729410Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5729701Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5729920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11411
2022-12-01T10:35:33.5730135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11412
2022-12-01T10:35:33.5730351Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11413
2022-12-01T10:35:33.5730563Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11414
2022-12-01T10:35:33.5730936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5731109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5731469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5731658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5732017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5732188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5732560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5732748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5733107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5733282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5733636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5733822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5734178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5734551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5734939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5735131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5735360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5735590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5735813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5736022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5736122Z ok (3.965s)
2022-12-01T10:35:33.5736143Z 
2022-12-01T10:35:33.5736411Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5736520Z Ran 1 test in 3.965s
2022-12-01T10:35:33.5736540Z 
2022-12-01T10:35:33.5736630Z OK
2022-12-01T10:35:33.5736650Z 
2022-12-01T10:35:33.5736771Z Generating XML reports...
2022-12-01T10:35:33.5737277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103157.xml
2022-12-01T10:35:33.5737660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5737880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5738260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5738449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5738469Z 
2022-12-01T10:35:33.5738577Z Running tests...
2022-12-01T10:35:33.5738839Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5739148Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5739397Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5739619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11594
2022-12-01T10:35:33.5739817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11595
2022-12-01T10:35:33.5740035Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11596
2022-12-01T10:35:33.5740245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11597
2022-12-01T10:35:33.5740614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5740787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5741160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5741350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5741707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5741879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5742232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5742423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5742780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5742950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5743313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5743498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5743863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5744035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5744382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5744570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5744800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5745027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5745246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5745471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5745573Z ok (3.941s)
2022-12-01T10:35:33.5745594Z 
2022-12-01T10:35:33.5745858Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5745968Z Ran 1 test in 3.942s
2022-12-01T10:35:33.5745989Z 
2022-12-01T10:35:33.5746111Z OK
2022-12-01T10:35:33.5746133Z 
2022-12-01T10:35:33.5746262Z Generating XML reports...
2022-12-01T10:35:33.5746697Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103203.xml
2022-12-01T10:35:33.5747115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5747289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5747663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5747852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5747872Z 
2022-12-01T10:35:33.5747978Z Running tests...
2022-12-01T10:35:33.5748222Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5748529Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5748788Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5749004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11777
2022-12-01T10:35:33.5749223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11778
2022-12-01T10:35:33.5749436Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11779
2022-12-01T10:35:33.5749643Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11780
2022-12-01T10:35:33.5750011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5750183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5750541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5750735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5751094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5751269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5751636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5751823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5752179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5752352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5752703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5752890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5753247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5753418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5753792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5753978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5754208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5754432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5754655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5754857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5754960Z ok (5.757s)
2022-12-01T10:35:33.5754981Z 
2022-12-01T10:35:33.5755306Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5755425Z Ran 1 test in 5.757s
2022-12-01T10:35:33.5755487Z 
2022-12-01T10:35:33.5755580Z OK
2022-12-01T10:35:33.5755600Z 
2022-12-01T10:35:33.5755722Z Generating XML reports...
2022-12-01T10:35:33.5756156Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103209.xml
2022-12-01T10:35:33.5756523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5756676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5757052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5757240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5757260Z 
2022-12-01T10:35:33.5757367Z Running tests...
2022-12-01T10:35:33.5757630Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5757938Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5758189Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5758408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11964
2022-12-01T10:35:33.5758609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11965
2022-12-01T10:35:33.5758825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11966
2022-12-01T10:35:33.5759035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11967
2022-12-01T10:35:33.5759406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5759580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5759961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5760154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5760519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5760692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5761046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5761233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5761595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5761766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5762136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5762321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5762684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5762856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5763211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5763396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5763626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5763853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5764124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5764354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5764454Z ok (3.966s)
2022-12-01T10:35:33.5764515Z 
2022-12-01T10:35:33.5764791Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5764901Z Ran 1 test in 3.966s
2022-12-01T10:35:33.5764921Z 
2022-12-01T10:35:33.5764995Z OK
2022-12-01T10:35:33.5765014Z 
2022-12-01T10:35:33.5765137Z Generating XML reports...
2022-12-01T10:35:33.5765567Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103217.xml
2022-12-01T10:35:33.5765932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5766104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5766479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5766667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5766687Z 
2022-12-01T10:35:33.5766798Z Running tests...
2022-12-01T10:35:33.5767044Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5767356Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5767603Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5767817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12147
2022-12-01T10:35:33.5768031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12148
2022-12-01T10:35:33.5768243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12149
2022-12-01T10:35:33.5768457Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12150
2022-12-01T10:35:33.5768826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5768999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5769366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5769555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5769912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5770084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5770459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5770644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5771006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5771178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5771543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5771728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5772083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5772252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5772621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5772808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5773036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5773312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5773545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5773803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5773904Z ok (4.056s)
2022-12-01T10:35:33.5773924Z 
2022-12-01T10:35:33.5774189Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5774299Z Ran 1 test in 4.056s
2022-12-01T10:35:33.5774529Z 
2022-12-01T10:35:33.5774628Z OK
2022-12-01T10:35:33.5774648Z 
2022-12-01T10:35:33.5774776Z Generating XML reports...
2022-12-01T10:35:33.5775215Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103224.xml
2022-12-01T10:35:33.5775579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5775743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5776121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5776314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5776334Z 
2022-12-01T10:35:33.5776442Z Running tests...
2022-12-01T10:35:33.5776701Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5777008Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5777266Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5777483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12354
2022-12-01T10:35:33.5777681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12355
2022-12-01T10:35:33.5777900Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12356
2022-12-01T10:35:33.5778109Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12357
2022-12-01T10:35:33.5778486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5778658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5779031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5779222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5779581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5779751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5780106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5780296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5780652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5780824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5781188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5781374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5781730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5781901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5782259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5782528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5782769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5783052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5783271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5783495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5783595Z ok (6.086s)
2022-12-01T10:35:33.5783615Z 
2022-12-01T10:35:33.5783885Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5783996Z Ran 1 test in 6.087s
2022-12-01T10:35:33.5784016Z 
2022-12-01T10:35:33.5784090Z OK
2022-12-01T10:35:33.5784109Z 
2022-12-01T10:35:33.5784232Z Generating XML reports...
2022-12-01T10:35:33.5784663Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103230.xml
2022-12-01T10:35:33.5785027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5785204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5785577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5785765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5785785Z 
2022-12-01T10:35:33.5785893Z Running tests...
2022-12-01T10:35:33.5786137Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5786448Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5786688Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5786906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12565
2022-12-01T10:35:33.5787121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12566
2022-12-01T10:35:33.5787333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12567
2022-12-01T10:35:33.5787551Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12568
2022-12-01T10:35:33.5787920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5788096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5788458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5788649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5789012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5789188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5789557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5789747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5790105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5790277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5790627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5790814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5791168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5791388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5791774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5792004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5792233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5792459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5792684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5792893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5792994Z ok (3.947s)
2022-12-01T10:35:33.5793015Z 
2022-12-01T10:35:33.5793281Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5793391Z Ran 1 test in 3.947s
2022-12-01T10:35:33.5793411Z 
2022-12-01T10:35:33.5793503Z OK
2022-12-01T10:35:33.5793528Z 
2022-12-01T10:35:33.5793650Z Generating XML reports...
2022-12-01T10:35:33.5794082Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103238.xml
2022-12-01T10:35:33.5794454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5794609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5794989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5795179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5795199Z 
2022-12-01T10:35:33.5795307Z Running tests...
2022-12-01T10:35:33.5795569Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5795878Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5796123Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5796339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12748
2022-12-01T10:35:33.5796542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12749
2022-12-01T10:35:33.5796756Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12750
2022-12-01T10:35:33.5796969Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12751
2022-12-01T10:35:33.5797341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5797513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5797887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5798079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5798442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5798618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5798970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5799154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5799513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5799684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5800060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5800245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5800649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5800825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5801231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5801421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5801648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5801874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5802096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5802320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5802420Z ok (3.911s)
2022-12-01T10:35:33.5802440Z 
2022-12-01T10:35:33.5802711Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5802820Z Ran 1 test in 3.911s
2022-12-01T10:35:33.5802841Z 
2022-12-01T10:35:33.5802918Z OK
2022-12-01T10:35:33.5802938Z 
2022-12-01T10:35:33.5803062Z Generating XML reports...
2022-12-01T10:35:33.5803495Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103244.xml
2022-12-01T10:35:33.5803859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5804032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5804404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5804595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5804615Z 
2022-12-01T10:35:33.5804724Z Running tests...
2022-12-01T10:35:33.5804974Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5805285Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5805538Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5805756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12931
2022-12-01T10:35:33.5805969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12932
2022-12-01T10:35:33.5806183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12933
2022-12-01T10:35:33.5806393Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12934
2022-12-01T10:35:33.5806760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5806933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5807297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5807485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5807850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5808021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5808388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5808575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5808932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5809101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5809497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5809692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5810107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5810278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5810640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5810825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5811052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5811280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5811499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5811711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5811812Z ok (5.753s)
2022-12-01T10:35:33.5811836Z 
2022-12-01T10:35:33.5812101Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5812211Z Ran 1 test in 5.754s
2022-12-01T10:35:33.5812230Z 
2022-12-01T10:35:33.5812322Z OK
2022-12-01T10:35:33.5812341Z 
2022-12-01T10:35:33.5812463Z Generating XML reports...
2022-12-01T10:35:33.5812893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103250.xml
2022-12-01T10:35:33.5813257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5813412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5813787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5813974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5813994Z 
2022-12-01T10:35:33.5814101Z Running tests...
2022-12-01T10:35:33.5814573Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5814900Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5815145Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5815363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13118
2022-12-01T10:35:33.5815562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13119
2022-12-01T10:35:33.5815776Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13120
2022-12-01T10:35:33.5815988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13121
2022-12-01T10:35:33.5816360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5816535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5816919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5817109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5817475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5817646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5817998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5818187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5818635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5818820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5819201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5819447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5819808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5819977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5820330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5820512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5820739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5820970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5821195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5821421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5821521Z ok (4.076s)
2022-12-01T10:35:33.5821542Z 
2022-12-01T10:35:33.5821806Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5821916Z Ran 1 test in 4.076s
2022-12-01T10:35:33.5821935Z 
2022-12-01T10:35:33.5822009Z OK
2022-12-01T10:35:33.5822029Z 
2022-12-01T10:35:33.5822153Z Generating XML reports...
2022-12-01T10:35:33.5822583Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103258.xml
2022-12-01T10:35:33.5822949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5823125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5823499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5823693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5823713Z 
2022-12-01T10:35:33.5823820Z Running tests...
2022-12-01T10:35:33.5824067Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5824377Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5824639Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5824855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13301
2022-12-01T10:35:33.5825072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13302
2022-12-01T10:35:33.5825288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13303
2022-12-01T10:35:33.5825503Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13304
2022-12-01T10:35:33.5825876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5826048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5826408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5826596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5826958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5827129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5827498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5827735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5828105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5828325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5828687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5828871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5829225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5829396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5829807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5829999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5830228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5830458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5830684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5830891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5830991Z ok (3.966s)
2022-12-01T10:35:33.5831011Z 
2022-12-01T10:35:33.5831274Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5831382Z Ran 1 test in 3.967s
2022-12-01T10:35:33.5831402Z 
2022-12-01T10:35:33.5831493Z OK
2022-12-01T10:35:33.5831512Z 
2022-12-01T10:35:33.5831633Z Generating XML reports...
2022-12-01T10:35:33.5832067Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103305.xml
2022-12-01T10:35:33.5832435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5832595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5832969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5833158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5833178Z 
2022-12-01T10:35:33.5833286Z Running tests...
2022-12-01T10:35:33.5833548Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5833862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5834103Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5834326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13484
2022-12-01T10:35:33.5834523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13485
2022-12-01T10:35:33.5834738Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13486
2022-12-01T10:35:33.5834951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13487
2022-12-01T10:35:33.5835324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5835497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5835872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5836066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5836417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5836645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5837027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5837261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5837621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5837791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5838157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5838342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5838706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5838858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5839226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5839416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5839646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5839872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5840089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5840314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5840414Z ok (4.558s)
2022-12-01T10:35:33.5840434Z 
2022-12-01T10:35:33.5840682Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5840791Z Ran 1 test in 4.558s
2022-12-01T10:35:33.5840811Z 
2022-12-01T10:35:33.5840902Z OK
2022-12-01T10:35:33.5840925Z 
2022-12-01T10:35:33.5841092Z Generating XML reports...
2022-12-01T10:35:33.5841529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103311.xml
2022-12-01T10:35:33.5841897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5842068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5842437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5842625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5842646Z 
2022-12-01T10:35:33.5842736Z Running tests...
2022-12-01T10:35:33.5842997Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5843305Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5843559Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5843780Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13691
2022-12-01T10:35:33.5844000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13692
2022-12-01T10:35:33.5844214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13693
2022-12-01T10:35:33.5844427Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13694
2022-12-01T10:35:33.5844777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5844955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5845333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5845573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5845951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5846174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5846547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5846735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5847093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5847249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5847618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5847803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5848164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5848335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5848709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5848899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5849126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5849335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5849562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5849785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5849885Z ok (7.476s)
2022-12-01T10:35:33.5849905Z 
2022-12-01T10:35:33.5850173Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5850285Z Ran 1 test in 7.476s
2022-12-01T10:35:33.5850304Z 
2022-12-01T10:35:33.5850399Z OK
2022-12-01T10:35:33.5850418Z 
2022-12-01T10:35:33.5850541Z Generating XML reports...
2022-12-01T10:35:33.5850955Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103318.xml
2022-12-01T10:35:33.5851323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5851497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5851872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5852060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5852080Z 
2022-12-01T10:35:33.5852187Z Running tests...
2022-12-01T10:35:33.5852454Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5852766Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5853032Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5853232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13902
2022-12-01T10:35:33.5853451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13903
2022-12-01T10:35:33.5853661Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13904
2022-12-01T10:35:33.5853874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13905
2022-12-01T10:35:33.5854244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5854633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5855106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5855307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5855715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5855891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5856261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5856448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5856806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5856978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5857356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5857541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5857906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5858060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5858431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5858619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5858848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5859079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5859302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5859531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5859631Z ok (4.064s)
2022-12-01T10:35:33.5859655Z 
2022-12-01T10:35:33.5859905Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5860016Z Ran 1 test in 4.064s
2022-12-01T10:35:33.5860035Z 
2022-12-01T10:35:33.5860126Z OK
2022-12-01T10:35:33.5860145Z 
2022-12-01T10:35:33.5860267Z Generating XML reports...
2022-12-01T10:35:33.5860702Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103328.xml
2022-12-01T10:35:33.5861068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5861242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5861618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5861809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5861829Z 
2022-12-01T10:35:33.5861918Z Running tests...
2022-12-01T10:35:33.5862187Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5862498Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5862741Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5862954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14089
2022-12-01T10:35:33.5863168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14090
2022-12-01T10:35:33.5863380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14091
2022-12-01T10:35:33.5863589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14092
2022-12-01T10:35:33.5863993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5864176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5864648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5864838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5865198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5865369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5865740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5865928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5866286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5866441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5866805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5866993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5867358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5867527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5867894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5868080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5868306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5868518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5868738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5868963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5869063Z ok (3.951s)
2022-12-01T10:35:33.5869083Z 
2022-12-01T10:35:33.5869346Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5869457Z Ran 1 test in 3.952s
2022-12-01T10:35:33.5869477Z 
2022-12-01T10:35:33.5869572Z OK
2022-12-01T10:35:33.5869592Z 
2022-12-01T10:35:33.5869712Z Generating XML reports...
2022-12-01T10:35:33.5870129Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103334.xml
2022-12-01T10:35:33.5870495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5870674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5871053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5871246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5871265Z 
2022-12-01T10:35:33.5871371Z Running tests...
2022-12-01T10:35:33.5871631Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5871939Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5872195Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5872397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14272
2022-12-01T10:35:33.5872615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14273
2022-12-01T10:35:33.5872877Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14274
2022-12-01T10:35:33.5873099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14275
2022-12-01T10:35:33.5873472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5873693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5874069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5874256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5874602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5874774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5875141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5875332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5875691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5875868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5876234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5876422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5876779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5876933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5877308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5877496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5877725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5877958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5878181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5878406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5878507Z ok (5.793s)
2022-12-01T10:35:33.5878527Z 
2022-12-01T10:35:33.5878777Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5878888Z Ran 1 test in 5.793s
2022-12-01T10:35:33.5878908Z 
2022-12-01T10:35:33.5878998Z OK
2022-12-01T10:35:33.5879017Z 
2022-12-01T10:35:33.5879138Z Generating XML reports...
2022-12-01T10:35:33.5879570Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103340.xml
2022-12-01T10:35:33.5879937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5880113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5880486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5880676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5880696Z 
2022-12-01T10:35:33.5880786Z Running tests...
2022-12-01T10:35:33.5881045Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5881355Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5881596Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5881876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14459
2022-12-01T10:35:33.5882100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14460
2022-12-01T10:35:33.5882314Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14461
2022-12-01T10:35:33.5882572Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14462
2022-12-01T10:35:33.5882928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5883103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5883484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5883673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5884032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5884208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5884578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5884769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5885125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5885278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5885651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5885836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5886190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5886360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5886731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5886923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5887151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5887360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5887587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5887811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5887913Z ok (4.057s)
2022-12-01T10:35:33.5887933Z 
2022-12-01T10:35:33.5888198Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5888308Z Ran 1 test in 4.057s
2022-12-01T10:35:33.5888328Z 
2022-12-01T10:35:33.5888419Z OK
2022-12-01T10:35:33.5888442Z 
2022-12-01T10:35:33.5888565Z Generating XML reports...
2022-12-01T10:35:33.5888978Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103348.xml
2022-12-01T10:35:33.5889351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5889526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5889906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5890094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5890114Z 
2022-12-01T10:35:33.5890221Z Running tests...
2022-12-01T10:35:33.5890483Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5890793Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5891089Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5891296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14642
2022-12-01T10:35:33.5891558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14643
2022-12-01T10:35:33.5891775Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14644
2022-12-01T10:35:33.5891986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14645
2022-12-01T10:35:33.5892356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5892534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5892911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5893099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5893447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5893625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5893999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5894182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5894819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5894995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5895377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5895563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5895934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5896091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5896463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5896648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5896877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5897104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5897327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5897553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5897653Z ok (4.260s)
2022-12-01T10:35:33.5897674Z 
2022-12-01T10:35:33.5897925Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5898036Z Ran 1 test in 4.260s
2022-12-01T10:35:33.5898056Z 
2022-12-01T10:35:33.5898151Z OK
2022-12-01T10:35:33.5898171Z 
2022-12-01T10:35:33.5898294Z Generating XML reports...
2022-12-01T10:35:33.5898724Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103354.xml
2022-12-01T10:35:33.5899089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5899263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5899638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5899828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5899847Z 
2022-12-01T10:35:33.5899938Z Running tests...
2022-12-01T10:35:33.5900281Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5900608Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5900921Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5901142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14849
2022-12-01T10:35:33.5901358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14850
2022-12-01T10:35:33.5901573Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14851
2022-12-01T10:35:33.5901785Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14852
2022-12-01T10:35:33.5902139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5902313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5902691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5902882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5903255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5903429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5903804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5903991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5904350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5904506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5904878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5905063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5905427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5905601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5905974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5906159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5906389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5906598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5906826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5907056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5907159Z ok (6.570s)
2022-12-01T10:35:33.5907184Z 
2022-12-01T10:35:33.5907450Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5907562Z Ran 1 test in 6.570s
2022-12-01T10:35:33.5907582Z 
2022-12-01T10:35:33.5907673Z OK
2022-12-01T10:35:33.5907693Z 
2022-12-01T10:35:33.5907815Z Generating XML reports...
2022-12-01T10:35:33.5908224Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103401.xml
2022-12-01T10:35:33.5908590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5908765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5909139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5909381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5909404Z 
2022-12-01T10:35:33.5909517Z Running tests...
2022-12-01T10:35:33.5909827Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5910141Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5910378Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5910579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15060
2022-12-01T10:35:33.5910796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15061
2022-12-01T10:35:33.5911010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15062
2022-12-01T10:35:33.5911223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15063
2022-12-01T10:35:33.5911594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5911768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5912153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5912342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5912684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5912858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5913230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5913416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5913775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5913951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5914325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5914517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5914874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5915028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5915403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5915589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5915818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5916048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5916274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5916505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5917061Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5917608Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5918203Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5918782Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5918884Z ok (4.053s)
2022-12-01T10:35:33.5918905Z 
2022-12-01T10:35:33.5919157Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5919269Z Ran 1 test in 4.053s
2022-12-01T10:35:33.5919289Z 
2022-12-01T10:35:33.5919380Z OK
2022-12-01T10:35:33.5919400Z 
2022-12-01T10:35:33.5919523Z Generating XML reports...
2022-12-01T10:35:33.5919959Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103410.xml
2022-12-01T10:35:33.5920333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5920513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5920894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5921065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5921101Z 
2022-12-01T10:35:33.5921192Z Running tests...
2022-12-01T10:35:33.5921453Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5921765Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5922026Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5922248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15255
2022-12-01T10:35:33.5922465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15256
2022-12-01T10:35:33.5922682Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15257
2022-12-01T10:35:33.5922892Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15258
2022-12-01T10:35:33.5923244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5923420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5923797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5923986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5924347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5924519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5924889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5925084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5925423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5925598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5925968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5926154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5926520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5926750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5927130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5927365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5927595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5927806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5928027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5928254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5928804Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5929349Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5929939Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5930471Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5931003Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5931538Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5932071Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5932598Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2022-12-01T10:35:33.5932707Z ok (4.161s)
2022-12-01T10:35:33.5932727Z 
2022-12-01T10:35:33.5932999Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5933112Z Ran 1 test in 4.161s
2022-12-01T10:35:33.5933131Z 
2022-12-01T10:35:33.5933223Z OK
2022-12-01T10:35:33.5933242Z 
2022-12-01T10:35:33.5933347Z Generating XML reports...
2022-12-01T10:35:33.5933784Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103416.xml
2022-12-01T10:35:33.5934203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5934605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5935006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5935283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5935304Z 
2022-12-01T10:35:33.5935414Z Running tests...
2022-12-01T10:35:33.5935683Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5935994Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5936224Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5936440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15474
2022-12-01T10:35:33.5936657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15475
2022-12-01T10:35:33.5936873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15476
2022-12-01T10:35:33.5937082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15477
2022-12-01T10:35:33.5937460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5937633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5938014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5938187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5938550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5938720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5939092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5939281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5939645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5939817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5940187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5940355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5940720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5940890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5941260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5941445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5941673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5941905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5942125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5942351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5942438Z ok (3.947s)
2022-12-01T10:35:33.5942458Z 
2022-12-01T10:35:33.5942724Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5942837Z Ran 1 test in 3.948s
2022-12-01T10:35:33.5942858Z 
2022-12-01T10:35:33.5942948Z OK
2022-12-01T10:35:33.5942967Z 
2022-12-01T10:35:33.5943090Z Generating XML reports...
2022-12-01T10:35:33.5943601Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103422.xml
2022-12-01T10:35:33.5943989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5944215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5944583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5944775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5944795Z 
2022-12-01T10:35:33.5944903Z Running tests...
2022-12-01T10:35:33.5945164Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5945469Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5945727Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5945947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15657
2022-12-01T10:35:33.5946163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15658
2022-12-01T10:35:33.5946383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15659
2022-12-01T10:35:33.5946575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15660
2022-12-01T10:35:33.5946948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5947122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5947499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5947686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5948048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5948221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5948593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5948764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5949134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5949304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5949671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5949854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5950216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5950393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5950759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5950948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5951160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5951386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5951612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5951831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5951934Z ok (5.652s)
2022-12-01T10:35:33.5951954Z 
2022-12-01T10:35:33.5952216Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5952326Z Ran 1 test in 5.652s
2022-12-01T10:35:33.5952396Z 
2022-12-01T10:35:33.5952494Z OK
2022-12-01T10:35:33.5952514Z 
2022-12-01T10:35:33.5952621Z Generating XML reports...
2022-12-01T10:35:33.5953055Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103429.xml
2022-12-01T10:35:33.5953478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5953653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5954031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5954224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5954244Z 
2022-12-01T10:35:33.5954351Z Running tests...
2022-12-01T10:35:33.5954613Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5954925Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5955152Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5955376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15844
2022-12-01T10:35:33.5955592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15845
2022-12-01T10:35:33.5955805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15846
2022-12-01T10:35:33.5956018Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15847
2022-12-01T10:35:33.5956390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5956563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5956939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5957114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5957476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5957653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5958026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5958215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5958573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5958744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5959122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5959295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5959651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5959826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5960198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5960385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5960614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5960842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5961065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5961291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5961423Z ok (3.959s)
2022-12-01T10:35:33.5961446Z 
2022-12-01T10:35:33.5961716Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5961827Z Ran 1 test in 3.959s
2022-12-01T10:35:33.5961888Z 
2022-12-01T10:35:33.5961983Z OK
2022-12-01T10:35:33.5962002Z 
2022-12-01T10:35:33.5962126Z Generating XML reports...
2022-12-01T10:35:33.5962558Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103437.xml
2022-12-01T10:35:33.5962928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5963101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5963458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5963648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5963668Z 
2022-12-01T10:35:33.5963781Z Running tests...
2022-12-01T10:35:33.5964045Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5964356Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5964607Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5964824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16027
2022-12-01T10:35:33.5965039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16028
2022-12-01T10:35:33.5965252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16029
2022-12-01T10:35:33.5965445Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16030
2022-12-01T10:35:33.5965816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5965993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5966363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5966540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5966914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5967105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5967479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5967649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5968013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5968186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5968560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5968749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5969116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5969287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5969654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5969838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5970050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5970277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5970560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5970793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5970939Z ok (4.650s)
2022-12-01T10:35:33.5970959Z 
2022-12-01T10:35:33.5971229Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5971342Z Ran 1 test in 4.650s
2022-12-01T10:35:33.5971361Z 
2022-12-01T10:35:33.5971452Z OK
2022-12-01T10:35:33.5971472Z 
2022-12-01T10:35:33.5971578Z Generating XML reports...
2022-12-01T10:35:33.5972005Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103443.xml
2022-12-01T10:35:33.5972370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5972545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5972926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5973117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5973140Z 
2022-12-01T10:35:33.5973248Z Running tests...
2022-12-01T10:35:33.5973512Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5973823Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5974120Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2022-12-01T10:35:33.5974140Z 
2022-12-01T10:35:33.5974593Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5974714Z Ran 1 test in 0.001s
2022-12-01T10:35:33.5974734Z 
2022-12-01T10:35:33.5974842Z OK (skipped=1)
2022-12-01T10:35:33.5974861Z 
2022-12-01T10:35:33.5974981Z Generating XML reports...
2022-12-01T10:35:33.5975422Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103450.xml
2022-12-01T10:35:33.5975792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5975971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5976351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5976523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5976543Z 
2022-12-01T10:35:33.5976650Z Running tests...
2022-12-01T10:35:33.5976912Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5977220Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5977472Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5977695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16267
2022-12-01T10:35:33.5977912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16268
2022-12-01T10:35:33.5978132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16269
2022-12-01T10:35:33.5978326Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16270
2022-12-01T10:35:33.5978694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5978867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5979242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5979431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5979874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5980060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5980434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5980665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5981036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5981208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5981575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5981762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5982123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5982300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5982671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5982862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5983075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5983303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5983528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5983749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5983849Z ok (3.824s)
2022-12-01T10:35:33.5983869Z 
2022-12-01T10:35:33.5984133Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5984244Z Ran 1 test in 3.825s
2022-12-01T10:35:33.5984268Z 
2022-12-01T10:35:33.5984359Z OK
2022-12-01T10:35:33.5984379Z 
2022-12-01T10:35:33.5984485Z Generating XML reports...
2022-12-01T10:35:33.5984918Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103452.xml
2022-12-01T10:35:33.5985291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5985464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5985842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5986033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5986053Z 
2022-12-01T10:35:33.5986160Z Running tests...
2022-12-01T10:35:33.5986421Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5986731Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5986986Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2022-12-01T10:35:33.5987010Z 
2022-12-01T10:35:33.5987272Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5987382Z Ran 1 test in 0.001s
2022-12-01T10:35:33.5987402Z 
2022-12-01T10:35:33.5987507Z OK (skipped=1)
2022-12-01T10:35:33.5987526Z 
2022-12-01T10:35:33.5987647Z Generating XML reports...
2022-12-01T10:35:33.5988073Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103458.xml
2022-12-01T10:35:33.5988438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5988611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5989034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5989213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5989273Z 
2022-12-01T10:35:33.5989382Z Running tests...
2022-12-01T10:35:33.5989646Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5989955Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5990223Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5990441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16483
2022-12-01T10:35:33.5990657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16484
2022-12-01T10:35:33.5990870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16485
2022-12-01T10:35:33.5991107Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16486
2022-12-01T10:35:33.5991489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5991668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5992046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5992235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5992602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5992773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5993144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5993330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5993676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5993851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5994225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5994411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5994768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5994940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5995314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5995497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5995712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.5995940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.5996169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.5996395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.5996495Z ok (5.871s)
2022-12-01T10:35:33.5996515Z 
2022-12-01T10:35:33.5996783Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5996894Z Ran 1 test in 5.871s
2022-12-01T10:35:33.5996914Z 
2022-12-01T10:35:33.5997004Z OK
2022-12-01T10:35:33.5997025Z 
2022-12-01T10:35:33.5997130Z Generating XML reports...
2022-12-01T10:35:33.5997563Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103500.xml
2022-12-01T10:35:33.5997983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.5998160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.5998538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.5998775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.5998795Z 
2022-12-01T10:35:33.5998903Z Running tests...
2022-12-01T10:35:33.5999165Z ----------------------------------------------------------------------
2022-12-01T10:35:33.5999480Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.5999726Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.5999942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16850
2022-12-01T10:35:33.6000158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16851
2022-12-01T10:35:33.6000376Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16852
2022-12-01T10:35:33.6000593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16853
2022-12-01T10:35:33.6000969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6001142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6001507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6001663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6002033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6002221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6002599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6002786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6003151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6003324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6003704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6003890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6004231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6004403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6004777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6004967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6005195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:35:33.6005427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:35:33.6005652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:35:33.6005878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:35:33.6005961Z ok (3.940s)
2022-12-01T10:35:33.6005981Z 
2022-12-01T10:35:33.6006244Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6006354Z Ran 1 test in 3.940s
2022-12-01T10:35:33.6006374Z 
2022-12-01T10:35:33.6006464Z OK
2022-12-01T10:35:33.6006483Z 
2022-12-01T10:35:33.6006605Z Generating XML reports...
2022-12-01T10:35:33.6007134Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103508.xml
2022-12-01T10:35:33.6007518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6007741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6008121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6008294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6008313Z 
2022-12-01T10:35:33.6008423Z Running tests...
2022-12-01T10:35:33.6008687Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6008998Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6009175Z   test_forward_backward (__main__.ReducerTest) ... ok (0.007s)
2022-12-01T10:35:33.6009195Z 
2022-12-01T10:35:33.6009459Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6009570Z Ran 1 test in 0.012s
2022-12-01T10:35:33.6009589Z 
2022-12-01T10:35:33.6009687Z OK
2022-12-01T10:35:33.6009706Z 
2022-12-01T10:35:33.6009812Z Generating XML reports...
2022-12-01T10:35:33.6010206Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103514.xml
2022-12-01T10:35:33.6010572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6010747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6011125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6011313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6011333Z 
2022-12-01T10:35:33.6011440Z Running tests...
2022-12-01T10:35:33.6011705Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6012017Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6012868Z   test_forward_backward_optimizer (__main__.ReducerTest) ... [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:35:33.6012969Z ok (0.011s)
2022-12-01T10:35:33.6012989Z 
2022-12-01T10:35:33.6013231Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6013343Z Ran 1 test in 0.012s
2022-12-01T10:35:33.6013366Z 
2022-12-01T10:35:33.6013457Z OK
2022-12-01T10:35:33.6013476Z 
2022-12-01T10:35:33.6013600Z Generating XML reports...
2022-12-01T10:35:33.6013994Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103516.xml
2022-12-01T10:35:33.6014580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6014766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6015150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6015325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6015363Z 
2022-12-01T10:35:33.6015453Z Running tests...
2022-12-01T10:35:33.6015712Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6016102Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6016321Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... ok (0.007s)
2022-12-01T10:35:33.6016341Z 
2022-12-01T10:35:33.6016664Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6016776Z Ran 1 test in 0.012s
2022-12-01T10:35:33.6016796Z 
2022-12-01T10:35:33.6016887Z OK
2022-12-01T10:35:33.6016906Z 
2022-12-01T10:35:33.6017028Z Generating XML reports...
2022-12-01T10:35:33.6017402Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103519.xml
2022-12-01T10:35:33.6017766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6017940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6018315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6018508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6018529Z 
2022-12-01T10:35:33.6018639Z Running tests...
2022-12-01T10:35:33.6018900Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6019210Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6019378Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-12-01T10:35:33.6019416Z 
2022-12-01T10:35:33.6019655Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6019765Z Ran 1 test in 0.012s
2022-12-01T10:35:33.6019785Z 
2022-12-01T10:35:33.6019875Z OK
2022-12-01T10:35:33.6019895Z 
2022-12-01T10:35:33.6020016Z Generating XML reports...
2022-12-01T10:35:33.6020405Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103521.xml
2022-12-01T10:35:33.6020777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6020953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6021332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6021506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6021526Z 
2022-12-01T10:35:33.6021634Z Running tests...
2022-12-01T10:35:33.6021891Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6022200Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6022390Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... ok (0.007s)
2022-12-01T10:35:33.6022410Z 
2022-12-01T10:35:33.6022668Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6022779Z Ran 1 test in 0.012s
2022-12-01T10:35:33.6022798Z 
2022-12-01T10:35:33.6022892Z OK
2022-12-01T10:35:33.6022911Z 
2022-12-01T10:35:33.6023016Z Generating XML reports...
2022-12-01T10:35:33.6023402Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103523.xml
2022-12-01T10:35:33.6023772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6023946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6024323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6024511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6024531Z 
2022-12-01T10:35:33.6024638Z Running tests...
2022-12-01T10:35:33.6024895Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6025201Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6025423Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... ok (0.004s)
2022-12-01T10:35:33.6025445Z 
2022-12-01T10:35:33.6025713Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6025870Z Ran 1 test in 0.012s
2022-12-01T10:35:33.6025889Z 
2022-12-01T10:35:33.6025982Z OK
2022-12-01T10:35:33.6026001Z 
2022-12-01T10:35:33.6026123Z Generating XML reports...
2022-12-01T10:35:33.6026518Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103525.xml
2022-12-01T10:35:33.6026883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6027060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6027418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6027613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6027633Z 
2022-12-01T10:35:33.6027741Z Running tests...
2022-12-01T10:35:33.6027999Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6028311Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6028540Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.6028785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:35:33.6029188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:35:33.6029289Z ok (1.569s)
2022-12-01T10:35:33.6029309Z 
2022-12-01T10:35:33.6029551Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6029701Z Ran 1 test in 1.569s
2022-12-01T10:35:33.6029722Z 
2022-12-01T10:35:33.6029813Z OK
2022-12-01T10:35:33.6029837Z 
2022-12-01T10:35:33.6029960Z Generating XML reports...
2022-12-01T10:35:33.6030378Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221201103527.xml
2022-12-01T10:35:33.6030749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:35:33.6030923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:35:33.6031300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:35:33.6031471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:35:33.6031508Z 
2022-12-01T10:35:33.6031597Z Running tests...
2022-12-01T10:35:33.6031858Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6032168Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2022-12-01T10:35:33.6032409Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:35:33.6033155Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.540s)
2022-12-01T10:35:33.6033180Z 
2022-12-01T10:35:33.6033438Z ----------------------------------------------------------------------
2022-12-01T10:35:33.6033549Z Ran 1 test in 1.540s
2022-12-01T10:35:33.6033569Z 
2022-12-01T10:35:33.6033675Z OK (skipped=1)
2022-12-01T10:35:33.6033695Z 
2022-12-01T10:35:33.6033816Z Generating XML reports...
2022-12-01T10:35:33.6034190Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221201103531.xml
2022-12-01T10:35:33.6034211Z 
2022-12-01T10:35:33.6034686Z ##[endgroup]
2022-12-01T10:35:33.6035173Z FINISHED PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_8pmfbaxw)
2022-12-01T10:35:33.6035195Z 
2022-12-01T10:35:33.6035533Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-12-01 10:35:33.468101]
2022-12-01T10:35:33.6036026Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:35:33.468390]
2022-12-01T10:40:24.3256224Z 
2022-12-01T10:40:24.3258710Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision
2022-12-01T10:40:24.3259729Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_sowab58t)
2022-12-01T10:40:24.3260134Z 
2022-12-01T10:40:24.3260279Z Running tests...
2022-12-01T10:40:24.3266454Z ----------------------------------------------------------------------
2022-12-01T10:40:24.3267089Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-12-01T10:40:24.3267793Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17373
2022-12-01T10:40:24.3268289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17374
2022-12-01T10:40:24.3268903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3269759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3270781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3271675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3272455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3272870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3273493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3273925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3274392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3274918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3275543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3276259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3276793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3277329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3278603Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3279692Z   warnings.warn(
2022-12-01T10:40:24.3281367Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3282537Z   warnings.warn(
2022-12-01T10:40:24.3283639Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3284215Z   warnings.warn(
2022-12-01T10:40:24.3284984Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3285653Z   warnings.warn(
2022-12-01T10:40:24.3285951Z dist init r=1, world=2
2022-12-01T10:40:24.3286132Z dist init r=0, world=2
2022-12-01T10:40:24.3286426Z ok (5.428s)
2022-12-01T10:40:24.3287168Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17457
2022-12-01T10:40:24.3288039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17458
2022-12-01T10:40:24.3288743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3289239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3289996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3290564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3291698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3292451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3293527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3294386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3295851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3296685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3297910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3299244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3300149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3301040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3302694Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3303662Z   warnings.warn(
2022-12-01T10:40:24.3304966Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3305964Z   warnings.warn(
2022-12-01T10:40:24.3307278Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3308271Z   warnings.warn(
2022-12-01T10:40:24.3309583Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3310712Z   warnings.warn(
2022-12-01T10:40:24.3311866Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3313199Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3314433Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3315651Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3316266Z dist init r=0, world=2
2022-12-01T10:40:24.3316501Z dist init r=1, world=2
2022-12-01T10:40:24.3316740Z ok (5.713s)
2022-12-01T10:40:24.3317275Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17541
2022-12-01T10:40:24.3317879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17542
2022-12-01T10:40:24.3318495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3318951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3319530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3319984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3320567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3321013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3321585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3322033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3322485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3322979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3323620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3324311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3324839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3325309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3326161Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3326705Z   warnings.warn(
2022-12-01T10:40:24.3327454Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3328048Z   warnings.warn(
2022-12-01T10:40:24.3328807Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3329401Z   warnings.warn(
2022-12-01T10:40:24.3330156Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3330702Z   warnings.warn(
2022-12-01T10:40:24.3331567Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3332816Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3334062Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3335804Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3336419Z dist init r=0, world=2
2022-12-01T10:40:24.3336672Z dist init r=1, world=2
2022-12-01T10:40:24.3336894Z ok (5.814s)
2022-12-01T10:40:24.3337458Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17625
2022-12-01T10:40:24.3338114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17626
2022-12-01T10:40:24.3338708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3339162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3339738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3340206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3340774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3341228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3341804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3342249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3342705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3343362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3343895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3344530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3345145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3345638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3346575Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3347103Z   warnings.warn(
2022-12-01T10:40:24.3347854Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3348394Z   warnings.warn(
2022-12-01T10:40:24.3349162Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3349691Z   warnings.warn(
2022-12-01T10:40:24.3350443Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3350993Z   warnings.warn(
2022-12-01T10:40:24.3351878Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3353121Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3354336Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3355554Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3356154Z dist init r=1, world=2
2022-12-01T10:40:24.3356410Z dist init r=0, world=2
2022-12-01T10:40:24.3356631Z ok (5.713s)
2022-12-01T10:40:24.3357163Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17709
2022-12-01T10:40:24.3357789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17710
2022-12-01T10:40:24.3358400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3358831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3359405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3359877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3360455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3360880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3361506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3361979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3362483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3362978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3363636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3364324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3364873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3365346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3366218Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3366774Z   warnings.warn(
2022-12-01T10:40:24.3367503Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3368034Z   warnings.warn(
2022-12-01T10:40:24.3368793Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3369331Z   warnings.warn(
2022-12-01T10:40:24.3370059Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3370608Z   warnings.warn(
2022-12-01T10:40:24.3371490Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3372736Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3373960Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3375774Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3376359Z dist init r=1, world=2
2022-12-01T10:40:24.3376609Z dist init r=0, world=2
2022-12-01T10:40:24.3376848Z ok (5.814s)
2022-12-01T10:40:24.3377388Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17793
2022-12-01T10:40:24.3378121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17794
2022-12-01T10:40:24.3378746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3379267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3379826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3380295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3380872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3381317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3381873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3382340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3382798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3383442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3383980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3384629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3385149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3385599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3386460Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3387008Z   warnings.warn(
2022-12-01T10:40:24.3387760Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3388287Z   warnings.warn(
2022-12-01T10:40:24.3389047Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3389596Z   warnings.warn(
2022-12-01T10:40:24.3390359Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3390881Z   warnings.warn(
2022-12-01T10:40:24.3391765Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3393006Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3394288Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3395523Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3396188Z dist init r=1, world=2
2022-12-01T10:40:24.3396418Z dist init r=0, world=2
2022-12-01T10:40:24.3397171Z ok (5.914s)
2022-12-01T10:40:24.3397715Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17877
2022-12-01T10:40:24.3398307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17878
2022-12-01T10:40:24.3398923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3399378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3399957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3400415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3400991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3401434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3401987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3402451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3402904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3403404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3404041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3404728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3405250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3405719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3406567Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3407112Z   warnings.warn(
2022-12-01T10:40:24.3407861Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3408396Z   warnings.warn(
2022-12-01T10:40:24.3409137Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3409675Z   warnings.warn(
2022-12-01T10:40:24.3410419Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3410959Z   warnings.warn(
2022-12-01T10:40:24.3411877Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3413137Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3414809Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3416100Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3416707Z dist init r=0, world=2
2022-12-01T10:40:24.3416962Z dist init r=1, world=2
2022-12-01T10:40:24.3417182Z ok (5.914s)
2022-12-01T10:40:24.3417744Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17961
2022-12-01T10:40:24.3418388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17962
2022-12-01T10:40:24.3418981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3419430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3420005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3420472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3421029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3421474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3422044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3422488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3422940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3423437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3424094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3424768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3425289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3425756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3426621Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3427151Z   warnings.warn(
2022-12-01T10:40:24.3427898Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3428436Z   warnings.warn(
2022-12-01T10:40:24.3429290Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3429884Z   warnings.warn(
2022-12-01T10:40:24.3430639Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3431187Z   warnings.warn(
2022-12-01T10:40:24.3432064Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3433279Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3434516Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3435738Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3436342Z dist init r=1, world=2
2022-12-01T10:40:24.3436597Z dist init r=0, world=2
2022-12-01T10:40:24.3436817Z ok (5.814s)
2022-12-01T10:40:24.3437345Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18045
2022-12-01T10:40:24.3437961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18046
2022-12-01T10:40:24.3438567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3438999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3439571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3440038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3440618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3441045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3441612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3442075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3442510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3443006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3443657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3444340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3444893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3445369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3446285Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3446832Z   warnings.warn(
2022-12-01T10:40:24.3447556Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3448091Z   warnings.warn(
2022-12-01T10:40:24.3448854Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3449397Z   warnings.warn(
2022-12-01T10:40:24.3450128Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3450681Z   warnings.warn(
2022-12-01T10:40:24.3451560Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3452804Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3454018Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3455911Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3456521Z dist init r=0, world=2
2022-12-01T10:40:24.3456771Z dist init r=1, world=2
2022-12-01T10:40:24.3457007Z ok (5.714s)
2022-12-01T10:40:24.3457535Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18129
2022-12-01T10:40:24.3458157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18130
2022-12-01T10:40:24.3458770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3459222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3459779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3460247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3460821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3461245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3461908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3462386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3462907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3463383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3464041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3464772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3465296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3465744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3466604Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3467159Z   warnings.warn(
2022-12-01T10:40:24.3467908Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3468430Z   warnings.warn(
2022-12-01T10:40:24.3469189Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3469735Z   warnings.warn(
2022-12-01T10:40:24.3470486Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3471008Z   warnings.warn(
2022-12-01T10:40:24.3471893Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3473134Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3474353Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3475575Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3476173Z dist init r=1, world=2
2022-12-01T10:40:24.3476405Z dist init r=0, world=2
2022-12-01T10:40:24.3476644Z ok (5.713s)
2022-12-01T10:40:24.3477155Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18213
2022-12-01T10:40:24.3477744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18214
2022-12-01T10:40:24.3478407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3478867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3479505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3479957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3480537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3480977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3481527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3481990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3482450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3482942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3483584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3484271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3484793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3485263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3486106Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3486650Z   warnings.warn(
2022-12-01T10:40:24.3487398Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3487938Z   warnings.warn(
2022-12-01T10:40:24.3488683Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3489233Z   warnings.warn(
2022-12-01T10:40:24.3489985Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3490519Z   warnings.warn(
2022-12-01T10:40:24.3491387Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3492625Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3493854Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3495595Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3496276Z dist init r=1, world=2
2022-12-01T10:40:24.3496510Z dist init r=0, world=2
2022-12-01T10:40:24.3496748Z ok (5.813s)
2022-12-01T10:40:24.3497291Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18297
2022-12-01T10:40:24.3497916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18298
2022-12-01T10:40:24.3498511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3498961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3499539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3499995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3500569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3501014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3501579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3502024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3502475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3502978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3503634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3504305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3504828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3505298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3506144Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3506689Z   warnings.warn(
2022-12-01T10:40:24.3507434Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3507965Z   warnings.warn(
2022-12-01T10:40:24.3508707Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3509244Z   warnings.warn(
2022-12-01T10:40:24.3509989Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3510529Z   warnings.warn(
2022-12-01T10:40:24.3511450Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3512709Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3513989Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3515210Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3515815Z dist init r=1, world=2
2022-12-01T10:40:24.3516066Z dist init r=0, world=2
2022-12-01T10:40:24.3516289Z ok (5.814s)
2022-12-01T10:40:24.3516800Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18381
2022-12-01T10:40:24.3517398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18382
2022-12-01T10:40:24.3518006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3518440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3519014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3519484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3520047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3520495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3521062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3521521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3521955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3522444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3523098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3523771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3524289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3524764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3525626Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3526144Z   warnings.warn(
2022-12-01T10:40:24.3526890Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3527422Z   warnings.warn(
2022-12-01T10:40:24.3528234Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3528812Z   warnings.warn(
2022-12-01T10:40:24.3529578Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3530119Z   warnings.warn(
2022-12-01T10:40:24.3530995Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3532231Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3533446Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3534899Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3535505Z dist init r=1, world=2
2022-12-01T10:40:24.3535753Z dist init r=0, world=2
2022-12-01T10:40:24.3535993Z ok (5.814s)
2022-12-01T10:40:24.3536517Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18465
2022-12-01T10:40:24.3537149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18466
2022-12-01T10:40:24.3537761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3538193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3538763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3539230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3539805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3540234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3540799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3541268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3541718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3542199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3542853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3543538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3544040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3544586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3545462Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3546071Z   warnings.warn(
2022-12-01T10:40:24.3546802Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3547343Z   warnings.warn(
2022-12-01T10:40:24.3548102Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3548644Z   warnings.warn(
2022-12-01T10:40:24.3549377Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3549916Z   warnings.warn(
2022-12-01T10:40:24.3550795Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3552034Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3553257Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3554491Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3555075Z dist init r=0, world=2
2022-12-01T10:40:24.3555323Z dist init r=1, world=2
2022-12-01T10:40:24.3555560Z ok (5.814s)
2022-12-01T10:40:24.3556049Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18549
2022-12-01T10:40:24.3556649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18550
2022-12-01T10:40:24.3557257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3557707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3558261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3558723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3559296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3559734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3560283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3560794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3561254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3561781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3562444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3563132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3563651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3564103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3565005Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3565600Z   warnings.warn(
2022-12-01T10:40:24.3566354Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3566875Z   warnings.warn(
2022-12-01T10:40:24.3567636Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3568180Z   warnings.warn(
2022-12-01T10:40:24.3568934Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3569453Z   warnings.warn(
2022-12-01T10:40:24.3570341Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3571585Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3572811Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3574035Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3574802Z dist init r=1, world=2
2022-12-01T10:40:24.3575035Z dist init r=0, world=2
2022-12-01T10:40:24.3575272Z ok (5.814s)
2022-12-01T10:40:24.3575813Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18633
2022-12-01T10:40:24.3576424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18634
2022-12-01T10:40:24.3577109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3577567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3578140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3578662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3579238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3579682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3580252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3580700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3581149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3581649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3582288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3582979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3583496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3583962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3584811Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3585353Z   warnings.warn(
2022-12-01T10:40:24.3586103Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3586641Z   warnings.warn(
2022-12-01T10:40:24.3587385Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3587926Z   warnings.warn(
2022-12-01T10:40:24.3588674Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3589216Z   warnings.warn(
2022-12-01T10:40:24.3590080Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3591315Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3592530Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3593803Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3594463Z dist init r=1, world=2
2022-12-01T10:40:24.3594712Z dist init r=0, world=2
2022-12-01T10:40:24.3594931Z ok (5.914s)
2022-12-01T10:40:24.3595442Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18717
2022-12-01T10:40:24.3596041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18718
2022-12-01T10:40:24.3596635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3597088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3597664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3598134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3598694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3599138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3599705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3600150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3600601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3601095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3601748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3602422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3602945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3603413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3604279Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3604808Z   warnings.warn(
2022-12-01T10:40:24.3605560Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3606093Z   warnings.warn(
2022-12-01T10:40:24.3606856Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3607390Z   warnings.warn(
2022-12-01T10:40:24.3608138Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3608679Z   warnings.warn(
2022-12-01T10:40:24.3609563Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3610835Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3612117Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3613347Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3613948Z dist init r=0, world=2
2022-12-01T10:40:24.3614202Z dist init r=1, world=2
2022-12-01T10:40:24.3614662Z ok (5.814s)
2022-12-01T10:40:24.3615180Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18801
2022-12-01T10:40:24.3615810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18802
2022-12-01T10:40:24.3616422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3616856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3617430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3617898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3618480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3618907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3619476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3619945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3620383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3621036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3621571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3622222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3622727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3623203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3624073Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3624613Z   warnings.warn(
2022-12-01T10:40:24.3625352Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3625882Z   warnings.warn(
2022-12-01T10:40:24.3626719Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3627273Z   warnings.warn(
2022-12-01T10:40:24.3628009Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3628628Z   warnings.warn(
2022-12-01T10:40:24.3629513Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3630750Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3631975Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3633186Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3633788Z dist init r=0, world=2
2022-12-01T10:40:24.3634037Z dist init r=1, world=2
2022-12-01T10:40:24.3634275Z ok (5.714s)
2022-12-01T10:40:24.3634774Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18885
2022-12-01T10:40:24.3635378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18886
2022-12-01T10:40:24.3635987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3636434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3636992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3637457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3638030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3638449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3639023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3639483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3639938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3640414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3641062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3641746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3642266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3642715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3643622Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3644212Z   warnings.warn(
2022-12-01T10:40:24.3644959Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3645474Z   warnings.warn(
2022-12-01T10:40:24.3646232Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3646777Z   warnings.warn(
2022-12-01T10:40:24.3647525Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3648042Z   warnings.warn(
2022-12-01T10:40:24.3648932Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3650164Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3651393Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3652620Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3653210Z dist init r=1, world=2
2022-12-01T10:40:24.3653459Z dist init r=0, world=2
2022-12-01T10:40:24.3653695Z ok (5.713s)
2022-12-01T10:40:24.3654234Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18969
2022-12-01T10:40:24.3654998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18970
2022-12-01T10:40:24.3655620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3656069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3656630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3657097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3657676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3658118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3658672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3659135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3659660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3660166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3660871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3661559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3662077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3662531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3663398Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3663947Z   warnings.warn(
2022-12-01T10:40:24.3664737Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3665282Z   warnings.warn(
2022-12-01T10:40:24.3666028Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3666573Z   warnings.warn(
2022-12-01T10:40:24.3667325Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3667859Z   warnings.warn(
2022-12-01T10:40:24.3668728Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3669968Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3671197Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3672415Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3673022Z dist init r=1, world=2
2022-12-01T10:40:24.3673254Z dist init r=0, world=2
2022-12-01T10:40:24.3673491Z ok (5.814s)
2022-12-01T10:40:24.3674008Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19053
2022-12-01T10:40:24.3674609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19054
2022-12-01T10:40:24.3675202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3675650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3676275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3676740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3677370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3677809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3678377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3678823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3679279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3679784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3680438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3681106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3681633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3682100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3682949Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3683493Z   warnings.warn(
2022-12-01T10:40:24.3684238Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3684776Z   warnings.warn(
2022-12-01T10:40:24.3685520Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3686065Z   warnings.warn(
2022-12-01T10:40:24.3686815Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3687349Z   warnings.warn(
2022-12-01T10:40:24.3688214Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3689443Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3690679Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3691955Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3692558Z dist init r=0, world=2
2022-12-01T10:40:24.3692809Z dist init r=1, world=2
2022-12-01T10:40:24.3693077Z ok (5.714s)
2022-12-01T10:40:24.3693623Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19137
2022-12-01T10:40:24.3694254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19138
2022-12-01T10:40:24.3695083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3695517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3696092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3696561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3697120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3697566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3698135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3698600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3699034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3699526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3700181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3700849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3701367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3701842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3702702Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3703229Z   warnings.warn(
2022-12-01T10:40:24.3703976Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3704510Z   warnings.warn(
2022-12-01T10:40:24.3705274Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3705805Z   warnings.warn(
2022-12-01T10:40:24.3706563Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3707102Z   warnings.warn(
2022-12-01T10:40:24.3707977Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3709293Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3710588Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3711803Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3712408Z dist init r=0, world=2
2022-12-01T10:40:24.3712658Z dist init r=1, world=2
2022-12-01T10:40:24.3712893Z ok (5.814s)
2022-12-01T10:40:24.3713393Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19221
2022-12-01T10:40:24.3714001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19222
2022-12-01T10:40:24.3714607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3715036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3715609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3716078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3716652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3717078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3717644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3718110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3718543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3719038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3719695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3720384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3720887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3721362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3722220Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3722769Z   warnings.warn(
2022-12-01T10:40:24.3723500Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3724030Z   warnings.warn(
2022-12-01T10:40:24.3724787Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3725336Z   warnings.warn(
2022-12-01T10:40:24.3726128Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3726717Z   warnings.warn(
2022-12-01T10:40:24.3727604Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3728825Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3730055Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3731271Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3731865Z dist init r=0, world=2
2022-12-01T10:40:24.3732114Z dist init r=1, world=2
2022-12-01T10:40:24.3732350Z ok (5.814s)
2022-12-01T10:40:24.3732875Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19305
2022-12-01T10:40:24.3733504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19306
2022-12-01T10:40:24.3734110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3734718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3735282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3735748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3736323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3736763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3737314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3737778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3738234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3738715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3739370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3740061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3740578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3741028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3741959Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3742521Z   warnings.warn(
2022-12-01T10:40:24.3743341Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3743853Z   warnings.warn(
2022-12-01T10:40:24.3744606Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3745152Z   warnings.warn(
2022-12-01T10:40:24.3745910Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3746427Z   warnings.warn(
2022-12-01T10:40:24.3747311Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3748557Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3749781Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3751013Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3751617Z dist init r=1, world=2
2022-12-01T10:40:24.3751849Z dist init r=0, world=2
2022-12-01T10:40:24.3752087Z ok (5.814s)
2022-12-01T10:40:24.3752602Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19389
2022-12-01T10:40:24.3753185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19390
2022-12-01T10:40:24.3753799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3754245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3754817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3755272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3755848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3756290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3756841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3757300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3757751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3758294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3758944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3759685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3760204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3760672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3761519Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3762054Z   warnings.warn(
2022-12-01T10:40:24.3762798Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3763379Z   warnings.warn(
2022-12-01T10:40:24.3764081Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3764658Z   warnings.warn(
2022-12-01T10:40:24.3765407Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3765934Z   warnings.warn(
2022-12-01T10:40:24.3766802Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3768037Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3769255Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3770484Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3771085Z dist init r=1, world=2
2022-12-01T10:40:24.3771317Z dist init r=0, world=2
2022-12-01T10:40:24.3771555Z ok (5.814s)
2022-12-01T10:40:24.3772113Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19473
2022-12-01T10:40:24.3772759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19474
2022-12-01T10:40:24.3773374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3773805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3774752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3775173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3775875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3776226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3776792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3777235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3777685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3778177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3778834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3779501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3780023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3780490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3781360Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3781893Z   warnings.warn(
2022-12-01T10:40:24.3782643Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3783190Z   warnings.warn(
2022-12-01T10:40:24.3783950Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3784477Z   warnings.warn(
2022-12-01T10:40:24.3785231Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3785776Z   warnings.warn(
2022-12-01T10:40:24.3786661Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3787915Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3789127Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3790350Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3791007Z dist init r=1, world=2
2022-12-01T10:40:24.3791298Z dist init r=0, world=2
2022-12-01T10:40:24.3791577Z ok (5.814s)
2022-12-01T10:40:24.3792062Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19557
2022-12-01T10:40:24.3792745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19558
2022-12-01T10:40:24.3793358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3793817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3794376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3794849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3795448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3795913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3796501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3796955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3797411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3797912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3798551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3799243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3799771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3800247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3801104Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3801656Z   warnings.warn(
2022-12-01T10:40:24.3802409Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3802947Z   warnings.warn(
2022-12-01T10:40:24.3803701Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3804249Z   warnings.warn(
2022-12-01T10:40:24.3805013Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3805572Z   warnings.warn(
2022-12-01T10:40:24.3806449Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3807755Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3808999Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3810287Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3810900Z dist init r=1, world=2
2022-12-01T10:40:24.3811153Z dist init r=0, world=2
2022-12-01T10:40:24.3811375Z ok (5.814s)
2022-12-01T10:40:24.3811941Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19641
2022-12-01T10:40:24.3812590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19642
2022-12-01T10:40:24.3813198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3813629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3814209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3814856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3815440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3815872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3816455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3816943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3817396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3818067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3818601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3819244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3819751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3820222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3821090Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3821636Z   warnings.warn(
2022-12-01T10:40:24.3822368Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3822910Z   warnings.warn(
2022-12-01T10:40:24.3823673Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3824224Z   warnings.warn(
2022-12-01T10:40:24.3825037Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3827531Z   warnings.warn(
2022-12-01T10:40:24.3828467Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3829736Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3830967Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3832190Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3832829Z dist init r=0, world=2
2022-12-01T10:40:24.3833038Z dist init r=1, world=2
2022-12-01T10:40:24.3833282Z ok (5.613s)
2022-12-01T10:40:24.3833814Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19725
2022-12-01T10:40:24.3834422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19726
2022-12-01T10:40:24.3835038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3835547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3836077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3836533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3837114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3837560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3838115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3838584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3839135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3839711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3840322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3840884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3841402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3841874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3842881Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3843378Z   warnings.warn(
2022-12-01T10:40:24.3844227Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3844742Z   warnings.warn(
2022-12-01T10:40:24.3845494Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3846046Z   warnings.warn(
2022-12-01T10:40:24.3846800Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3847350Z   warnings.warn(
2022-12-01T10:40:24.3848217Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3849469Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3850703Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3851922Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3852529Z dist init r=0, world=2
2022-12-01T10:40:24.3852782Z dist init r=1, world=2
2022-12-01T10:40:24.3853000Z ok (5.715s)
2022-12-01T10:40:24.3853556Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19809
2022-12-01T10:40:24.3854202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19810
2022-12-01T10:40:24.3855128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3855558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3856136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3856609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3857166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3857611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3858178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3858639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3859074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3859662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3860345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3861100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3861626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3862097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3862963Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3863490Z   warnings.warn(
2022-12-01T10:40:24.3864247Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3864837Z   warnings.warn(
2022-12-01T10:40:24.3865599Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3866125Z   warnings.warn(
2022-12-01T10:40:24.3866882Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3867427Z   warnings.warn(
2022-12-01T10:40:24.3868319Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3869568Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3870778Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3872015Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3872643Z dist init r=1, world=2
2022-12-01T10:40:24.3872876Z dist init r=0, world=2
2022-12-01T10:40:24.3873127Z ok (5.814s)
2022-12-01T10:40:24.3873648Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19893
2022-12-01T10:40:24.3874233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19894
2022-12-01T10:40:24.3874845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3875277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3875853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3876376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3876965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3877447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3878015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3878478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3878911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3879405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3880060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3880754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3881258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3881732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3882600Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3883157Z   warnings.warn(
2022-12-01T10:40:24.3883881Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3884415Z   warnings.warn(
2022-12-01T10:40:24.3885214Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3885737Z   warnings.warn(
2022-12-01T10:40:24.3886470Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3887003Z   warnings.warn(
2022-12-01T10:40:24.3887885Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3889129Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3890353Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3891583Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3892166Z dist init r=0, world=2
2022-12-01T10:40:24.3892467Z dist init r=1, world=2
2022-12-01T10:40:24.3892715Z ok (5.817s)
2022-12-01T10:40:24.3893254Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19977
2022-12-01T10:40:24.3893956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19978
2022-12-01T10:40:24.3894787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3895226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3895773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3896242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3896822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3897282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3897818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3898279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3898729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3899205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3899867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3900556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3901086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3901542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3902406Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3902948Z   warnings.warn(
2022-12-01T10:40:24.3903700Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3904219Z   warnings.warn(
2022-12-01T10:40:24.3904995Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3905528Z   warnings.warn(
2022-12-01T10:40:24.3906275Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3906792Z   warnings.warn(
2022-12-01T10:40:24.3907669Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3908973Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3910222Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3911528Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3912132Z dist init r=0, world=2
2022-12-01T10:40:24.3912364Z dist init r=1, world=2
2022-12-01T10:40:24.3912600Z ok (5.817s)
2022-12-01T10:40:24.3913132Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20061
2022-12-01T10:40:24.3913737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20062
2022-12-01T10:40:24.3914352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3914803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3915378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3915833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3916407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3916848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3917408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3917880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3918335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3918829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3919470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3920155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3920674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3921144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3921998Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3922547Z   warnings.warn(
2022-12-01T10:40:24.3923297Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3923835Z   warnings.warn(
2022-12-01T10:40:24.3924581Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3925126Z   warnings.warn(
2022-12-01T10:40:24.3925958Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3926511Z   warnings.warn(
2022-12-01T10:40:24.3927428Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3928665Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3929889Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3931119Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3931715Z dist init r=0, world=2
2022-12-01T10:40:24.3931950Z dist init r=1, world=2
2022-12-01T10:40:24.3932189Z ok (5.814s)
2022-12-01T10:40:24.3932741Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20145
2022-12-01T10:40:24.3933383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20146
2022-12-01T10:40:24.3933977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3934687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3935293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3935747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3936310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3936754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3937328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3937775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3938235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3938731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3939396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3940066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3940591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3941064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3941934Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3942543Z   warnings.warn(
2022-12-01T10:40:24.3943308Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3943914Z   warnings.warn(
2022-12-01T10:40:24.3944678Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3945204Z   warnings.warn(
2022-12-01T10:40:24.3945950Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3946492Z   warnings.warn(
2022-12-01T10:40:24.3947376Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3948585Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3949824Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3951063Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3951667Z dist init r=0, world=2
2022-12-01T10:40:24.3951919Z dist init r=1, world=2
2022-12-01T10:40:24.3952137Z ok (5.614s)
2022-12-01T10:40:24.3952659Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20229
2022-12-01T10:40:24.3953267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20230
2022-12-01T10:40:24.3953877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3954311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3954885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3955355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3955915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3956360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3956929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3957394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3957827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3958318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3959028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3959729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3960302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3960772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3961641Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3962187Z   warnings.warn(
2022-12-01T10:40:24.3962920Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3963456Z   warnings.warn(
2022-12-01T10:40:24.3964212Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3964810Z   warnings.warn(
2022-12-01T10:40:24.3965580Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3966132Z   warnings.warn(
2022-12-01T10:40:24.3966980Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3968207Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3969441Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3970652Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3971250Z dist init r=1, world=2
2022-12-01T10:40:24.3971499Z dist init r=0, world=2
2022-12-01T10:40:24.3971740Z ok (5.714s)
2022-12-01T10:40:24.3972276Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20313
2022-12-01T10:40:24.3972906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20314
2022-12-01T10:40:24.3973512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3973960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3974691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3975244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3975843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3976337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3976910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3977372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3977824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3978462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3978988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3979636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3980156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.3980613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.3981470Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3982014Z   warnings.warn(
2022-12-01T10:40:24.3982766Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.3983288Z   warnings.warn(
2022-12-01T10:40:24.3984046Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3984598Z   warnings.warn(
2022-12-01T10:40:24.3985343Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.3985863Z   warnings.warn(
2022-12-01T10:40:24.3986755Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3988018Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3989219Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3990441Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.3991027Z dist init r=0, world=2
2022-12-01T10:40:24.3991275Z dist init r=1, world=2
2022-12-01T10:40:24.3991511Z ok (5.714s)
2022-12-01T10:40:24.3992084Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20397
2022-12-01T10:40:24.3992735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20398
2022-12-01T10:40:24.3993342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3993788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3994344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3994818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3995397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.3995874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.3996444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.3996901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.3997340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.3997832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.3998468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3999148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.3999667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4000143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4000992Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4001540Z   warnings.warn(
2022-12-01T10:40:24.4002291Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4002830Z   warnings.warn(
2022-12-01T10:40:24.4003576Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4004118Z   warnings.warn(
2022-12-01T10:40:24.4004869Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4005405Z   warnings.warn(
2022-12-01T10:40:24.4006270Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4007505Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4008785Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4010063Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4010675Z dist init r=0, world=2
2022-12-01T10:40:24.4010908Z dist init r=1, world=2
2022-12-01T10:40:24.4011148Z ok (5.713s)
2022-12-01T10:40:24.4011701Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20481
2022-12-01T10:40:24.4012331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20482
2022-12-01T10:40:24.4012923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4013377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4013941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4014570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4015149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4015602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4016188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4016638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4017091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4017591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4018245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4018920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4019442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4019907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4020778Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4021311Z   warnings.warn(
2022-12-01T10:40:24.4022068Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4022602Z   warnings.warn(
2022-12-01T10:40:24.4023348Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4023889Z   warnings.warn(
2022-12-01T10:40:24.4024721Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4025283Z   warnings.warn(
2022-12-01T10:40:24.4026170Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4027469Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4028696Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4029920Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4030526Z dist init r=1, world=2
2022-12-01T10:40:24.4030780Z dist init r=0, world=2
2022-12-01T10:40:24.4031002Z ok (5.814s)
2022-12-01T10:40:24.4031519Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20565
2022-12-01T10:40:24.4032127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20566
2022-12-01T10:40:24.4032740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4033174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4033754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4034222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4034784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4035228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4035796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4036256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4036688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4037185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4037839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4038529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4039032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4039499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4040364Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4040884Z   warnings.warn(
2022-12-01T10:40:24.4041695Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4042289Z   warnings.warn(
2022-12-01T10:40:24.4043051Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4043578Z   warnings.warn(
2022-12-01T10:40:24.4044330Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4044867Z   warnings.warn(
2022-12-01T10:40:24.4045753Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4046995Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4048209Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4049448Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4050055Z dist init r=1, world=2
2022-12-01T10:40:24.4050307Z dist init r=0, world=2
2022-12-01T10:40:24.4050543Z ok (5.814s)
2022-12-01T10:40:24.4051076Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20649
2022-12-01T10:40:24.4051708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20650
2022-12-01T10:40:24.4052316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4052746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4053330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4053796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4054599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4055052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4055605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4056074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4056530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4057008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4057742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4058450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4059028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4059506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4060377Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4060919Z   warnings.warn(
2022-12-01T10:40:24.4061663Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4062207Z   warnings.warn(
2022-12-01T10:40:24.4062970Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4063520Z   warnings.warn(
2022-12-01T10:40:24.4064255Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4064830Z   warnings.warn(
2022-12-01T10:40:24.4065715Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4066962Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4068186Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4069413Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4070000Z dist init r=1, world=2
2022-12-01T10:40:24.4070252Z dist init r=0, world=2
2022-12-01T10:40:24.4070489Z ok (5.714s)
2022-12-01T10:40:24.4070994Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20733
2022-12-01T10:40:24.4071607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20734
2022-12-01T10:40:24.4072215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4072669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4073227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4073698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4074332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4074790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4075398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4075865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4076320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4076796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4077454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4078141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4078670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4079125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4079989Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4080529Z   warnings.warn(
2022-12-01T10:40:24.4081278Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4081795Z   warnings.warn(
2022-12-01T10:40:24.4082558Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4083109Z   warnings.warn(
2022-12-01T10:40:24.4083869Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4084390Z   warnings.warn(
2022-12-01T10:40:24.4085274Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4086514Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4087749Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4088984Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:40:24.4089586Z dist init r=0, world=2
2022-12-01T10:40:24.4089821Z dist init r=1, world=2
2022-12-01T10:40:24.4090057Z ok (5.713s)
2022-12-01T10:40:24.4090608Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20817
2022-12-01T10:40:24.4091181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20818
2022-12-01T10:40:24.4091845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4092297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4092872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4093327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4093903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4094567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4095152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4095605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4096063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4096558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4097198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4097888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4098412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4098883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4099738Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4100289Z   warnings.warn(
2022-12-01T10:40:24.4101042Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4101581Z   warnings.warn(
2022-12-01T10:40:24.4102324Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4102869Z   warnings.warn(
2022-12-01T10:40:24.4103625Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4104165Z   warnings.warn(
2022-12-01T10:40:24.4104392Z dist init r=1, world=2
2022-12-01T10:40:24.4104640Z dist init r=0, world=2
2022-12-01T10:40:24.4104881Z ok (5.313s)
2022-12-01T10:40:24.4105204Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-12-01T10:40:24.4105663Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2022-12-01T10:40:24.4106257Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20901
2022-12-01T10:40:24.4106835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20902
2022-12-01T10:40:24.4107426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4107955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4108554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4109083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4109667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4110123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4110684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4111135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4111615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4112122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4112810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4113455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4113982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4135053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4135983Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4136533Z   warnings.warn(
2022-12-01T10:40:24.4137303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4137840Z   warnings.warn(
2022-12-01T10:40:24.4138592Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4139110Z   warnings.warn(
2022-12-01T10:40:24.4139855Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4140370Z   warnings.warn(
2022-12-01T10:40:24.4140622Z dist init r=0, world=2
2022-12-01T10:40:24.4140876Z dist init r=1, world=2
2022-12-01T10:40:24.4141097Z ok (5.615s)
2022-12-01T10:40:24.4141581Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20985
2022-12-01T10:40:24.4142156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20986
2022-12-01T10:40:24.4142770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4143203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4143766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4144216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4144773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4145247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4145978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4146467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4146980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4147483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4148144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4148837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4149345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4149815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4150691Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4151243Z   warnings.warn(
2022-12-01T10:40:24.4151984Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4152520Z   warnings.warn(
2022-12-01T10:40:24.4153287Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4153835Z   warnings.warn(
2022-12-01T10:40:24.4154584Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4155137Z   warnings.warn(
2022-12-01T10:40:24.4155386Z dist init r=0, world=2
2022-12-01T10:40:24.4155618Z dist init r=1, world=2
2022-12-01T10:40:24.4155859Z ok (5.414s)
2022-12-01T10:40:24.4156324Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21069
2022-12-01T10:40:24.4156866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21070
2022-12-01T10:40:24.4157480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4157929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4158510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4158965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4159550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4159991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4160561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4161011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4161463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4161963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4162667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4163377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4163973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4164447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4165348Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4165893Z   warnings.warn(
2022-12-01T10:40:24.4166647Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4167185Z   warnings.warn(
2022-12-01T10:40:24.4167939Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4168486Z   warnings.warn(
2022-12-01T10:40:24.4169248Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4169792Z   warnings.warn(
2022-12-01T10:40:24.4170020Z dist init r=0, world=2
2022-12-01T10:40:24.4170285Z dist init r=1, world=2
2022-12-01T10:40:24.4170509Z ok (5.413s)
2022-12-01T10:40:24.4170973Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21153
2022-12-01T10:40:24.4171543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21154
2022-12-01T10:40:24.4172152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4172603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4173161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4173632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4174210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4174829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4175402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4175876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4176333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4176820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4177472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4178148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4178672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4179125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4180071Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4180630Z   warnings.warn(
2022-12-01T10:40:24.4181390Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4181984Z   warnings.warn(
2022-12-01T10:40:24.4182743Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4183265Z   warnings.warn(
2022-12-01T10:40:24.4183997Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4184515Z   warnings.warn(
2022-12-01T10:40:24.4184762Z dist init r=1, world=2
2022-12-01T10:40:24.4185015Z dist init r=0, world=2
2022-12-01T10:40:24.4185238Z ok (5.415s)
2022-12-01T10:40:24.4185721Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21237
2022-12-01T10:40:24.4186299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21238
2022-12-01T10:40:24.4186906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4187339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4187912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4188380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4188943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4189391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4189962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4190425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4190864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4191360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4192016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4192707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4193216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4193692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4194564Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4195092Z   warnings.warn(
2022-12-01T10:40:24.4195851Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4196388Z   warnings.warn(
2022-12-01T10:40:24.4197202Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4197741Z   warnings.warn(
2022-12-01T10:40:24.4198551Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4199093Z   warnings.warn(
2022-12-01T10:40:24.4199342Z dist init r=0, world=2
2022-12-01T10:40:24.4199573Z dist init r=1, world=2
2022-12-01T10:40:24.4199813Z ok (5.513s)
2022-12-01T10:40:24.4200279Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21321
2022-12-01T10:40:24.4200817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21322
2022-12-01T10:40:24.4201436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4201889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4202464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4202921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4203502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4203946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4204495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4204963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4205416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:40:24.4205921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4206556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4207248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:40:24.4207769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:40:24.4208243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4209087Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4209631Z   warnings.warn(
2022-12-01T10:40:24.4210394Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4210939Z   warnings.warn(
2022-12-01T10:40:24.4211685Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4212234Z   warnings.warn(
2022-12-01T10:40:24.4212997Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:40:24.4213540Z   warnings.warn(
2022-12-01T10:40:24.4213769Z dist init r=1, world=2
2022-12-01T10:40:24.4214020Z dist init r=0, world=2
2022-12-01T10:40:24.4214261Z ok (5.813s)
2022-12-01T10:40:24.4215039Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21405
2022-12-01T10:40:24.4215827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4216279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4216857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4217311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4217766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4218422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:40:24.4218933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4219798Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4220350Z   warnings.warn(
2022-12-01T10:40:24.4220602Z dist init r=0, world=1
2022-12-01T10:40:24.4220826Z ok (4.109s)
2022-12-01T10:40:24.4221304Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21447
2022-12-01T10:40:24.4222032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4222483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4223047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4223514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4223966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4224610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:40:24.4225138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4225998Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4226542Z   warnings.warn(
2022-12-01T10:40:24.4226774Z dist init r=0, world=1
2022-12-01T10:40:24.4227017Z ok (4.109s)
2022-12-01T10:40:24.4227513Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21489
2022-12-01T10:40:24.4228237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:40:24.4228691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:40:24.4229263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:40:24.4229733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:40:24.4230167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:40:24.4230822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:40:24.4231342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:40:24.4232248Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:40:24.4232821Z   warnings.warn(
2022-12-01T10:40:24.4233071Z dist init r=0, world=1
2022-12-01T10:40:24.4233313Z ok (4.109s)
2022-12-01T10:40:24.4233460Z 
2022-12-01T10:40:24.4233719Z ----------------------------------------------------------------------
2022-12-01T10:40:24.4234044Z Ran 52 tests in 287.415s
2022-12-01T10:40:24.4234204Z 
2022-12-01T10:40:24.4234310Z OK (skipped=1)
2022-12-01T10:40:24.4234462Z 
2022-12-01T10:40:24.4234586Z Generating XML reports...
2022-12-01T10:40:24.4235235Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221201103536.xml
2022-12-01T10:40:24.4236134Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221201103536.xml
2022-12-01T10:40:24.4236539Z 
2022-12-01T10:40:24.4236932Z ##[endgroup]
2022-12-01T10:40:24.4237547Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_sowab58t)
2022-12-01T10:40:24.4237935Z 
2022-12-01T10:40:24.4238229Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2022-12-01 10:40:24.327296]
2022-12-01T10:40:24.4238939Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:40:24.327571]
2022-12-01T10:44:42.6524014Z 
2022-12-01T10:44:42.6526840Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params
2022-12-01T10:44:42.6527894Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_eptkzh4h)
2022-12-01T10:44:42.6528354Z 
2022-12-01T10:44:42.6528551Z Running tests...
2022-12-01T10:44:42.6529538Z ----------------------------------------------------------------------
2022-12-01T10:44:42.6532688Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2022-12-01T10:44:42.6533528Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:44:42.6534053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21566
2022-12-01T10:44:42.6534961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21567
2022-12-01T10:44:42.6536503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6536998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6537920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6538388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6539213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6539693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6540481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6540931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6541888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6542435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6543375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6544155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6544824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6545315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6546625Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6547513Z   warnings.warn(
2022-12-01T10:44:42.6548694Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6549504Z   warnings.warn(
2022-12-01T10:44:42.6550297Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6550876Z   warnings.warn(
2022-12-01T10:44:42.6551675Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6552316Z   warnings.warn(
2022-12-01T10:44:42.6552741Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 222, in bad_backwards_hook
2022-12-01T10:44:42.6553197Z     with model.summon_full_params(model):
2022-12-01T10:44:42.6553644Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-12-01T10:44:42.6554278Z     return next(self.gen)
2022-12-01T10:44:42.6555240Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2921, in summon_full_params
2022-12-01T10:44:42.6556107Z     stack.enter_context(
2022-12-01T10:44:42.6556716Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-12-01T10:44:42.6557353Z     result = _cm_type.__enter__(cm)
2022-12-01T10:44:42.6557963Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-12-01T10:44:42.6558574Z     return next(self.gen)
2022-12-01T10:44:42.6559598Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2961, in _summon_full_params
2022-12-01T10:44:42.6560352Z     stack.enter_context(
2022-12-01T10:44:42.6560967Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-12-01T10:44:42.6561591Z     result = _cm_type.__enter__(cm)
2022-12-01T10:44:42.6562217Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-12-01T10:44:42.6562767Z     return next(self.gen)
2022-12-01T10:44:42.6563761Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2974, in _summon_full_params
2022-12-01T10:44:42.6564613Z     self._assert_state([TrainingState_.IDLE])
2022-12-01T10:44:42.6565652Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3582, in _assert_state
2022-12-01T10:44:42.6566371Z     traceback.print_stack()
2022-12-01T10:44:42.6566818Z dist init r=1, world=2
2022-12-01T10:44:42.6567380Z dist init r=0, world=2
2022-12-01T10:44:42.6567995Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-12-01T10:44:42.6568638Z   (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-12-01T10:44:42.6569369Z     (_fpw_module): Linear(in_features=2, out_features=1, bias=True)
2022-12-01T10:44:42.6569913Z   )
2022-12-01T10:44:42.6570257Z )
2022-12-01T10:44:42.6570896Z ERROR: expected to be in states [<TrainingState_.IDLE: 1>] but current state is TrainingState_.BACKWARD_PRE
2022-12-01T10:44:42.6571523Z ok (5.466s)
2022-12-01T10:44:42.6572276Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21649
2022-12-01T10:44:42.6573305Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21650
2022-12-01T10:44:42.6574746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6575597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6576619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6577443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6578466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6579214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6580216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6581070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6582105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6582675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6583351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6584048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6584572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6585026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6586238Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6586990Z   warnings.warn(
2022-12-01T10:44:42.6588099Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6588846Z   warnings.warn(
2022-12-01T10:44:42.6589584Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6590126Z   warnings.warn(
2022-12-01T10:44:42.6590980Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6591530Z   warnings.warn(
2022-12-01T10:44:42.6591851Z   File "<string>", line 1, in <module>
2022-12-01T10:44:42.6592222Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2022-12-01T10:44:42.6592588Z     exitcode = _main(fd, parent_sentinel)
2022-12-01T10:44:42.6592957Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2022-12-01T10:44:42.6593308Z     return self._bootstrap(parent_sentinel)
2022-12-01T10:44:42.6593697Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
2022-12-01T10:44:42.6594031Z     self.run()
2022-12-01T10:44:42.6594344Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2022-12-01T10:44:42.6594710Z     self._target(*self._args, **self._kwargs)
2022-12-01T10:44:42.6595228Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 785, in _run
2022-12-01T10:44:42.6595600Z     self.run_test(test_name, pipe)
2022-12-01T10:44:42.6596171Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 622, in run_test
2022-12-01T10:44:42.6596567Z     getattr(self, test_name)()
2022-12-01T10:44:42.6597082Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 503, in wrapper
2022-12-01T10:44:42.6597432Z     fn()
2022-12-01T10:44:42.6597914Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 145, in wrapper
2022-12-01T10:44:42.6598303Z     return func(*args, **kwargs)
2022-12-01T10:44:42.6598733Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 213, in test_cannot_summon_full_params_from_forward
2022-12-01T10:44:42.6599135Z     model(model)
2022-12-01T10:44:42.6599601Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
2022-12-01T10:44:42.6599972Z     return forward_call(*input, **kwargs)
2022-12-01T10:44:42.6600518Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2727, in forward
2022-12-01T10:44:42.6600966Z     output = self._fsdp_wrapped_module(*args, **kwargs)
2022-12-01T10:44:42.6601483Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
2022-12-01T10:44:42.6601852Z     return forward_call(*input, **kwargs)
2022-12-01T10:44:42.6602381Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flatten_params_wrapper.py", line 165, in forward
2022-12-01T10:44:42.6602798Z     return self.module(*inputs, **kwinputs)
2022-12-01T10:44:42.6603278Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
2022-12-01T10:44:42.6603660Z     return forward_call(*input, **kwargs)
2022-12-01T10:44:42.6604074Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py", line 206, in forward
2022-12-01T10:44:42.6604495Z     with fsdp_module.summon_full_params(fsdp_module):
2022-12-01T10:44:42.6604859Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-12-01T10:44:42.6605182Z     return next(self.gen)
2022-12-01T10:44:42.6605735Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2921, in summon_full_params
2022-12-01T10:44:42.6606126Z     stack.enter_context(
2022-12-01T10:44:42.6606466Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-12-01T10:44:42.6606806Z     result = _cm_type.__enter__(cm)
2022-12-01T10:44:42.6607147Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-12-01T10:44:42.6607450Z     return next(self.gen)
2022-12-01T10:44:42.6608060Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2961, in _summon_full_params
2022-12-01T10:44:42.6608477Z     stack.enter_context(
2022-12-01T10:44:42.6608797Z   File "/opt/conda/lib/python3.10/contextlib.py", line 492, in enter_context
2022-12-01T10:44:42.6609194Z     result = _cm_type.__enter__(cm)
2022-12-01T10:44:42.6609535Z   File "/opt/conda/lib/python3.10/contextlib.py", line 135, in __enter__
2022-12-01T10:44:42.6609836Z     return next(self.gen)
2022-12-01T10:44:42.6610385Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 2974, in _summon_full_params
2022-12-01T10:44:42.6610823Z     self._assert_state([TrainingState_.IDLE])
2022-12-01T10:44:42.6611392Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3582, in _assert_state
2022-12-01T10:44:42.6611785Z     traceback.print_stack()
2022-12-01T10:44:42.6612049Z dist init r=1, world=2
2022-12-01T10:44:42.6612298Z dist init r=0, world=2
2022-12-01T10:44:42.6612612Z Asserting FSDP instance is: FullyShardedDataParallel(
2022-12-01T10:44:42.6612982Z   (_fsdp_wrapped_module): FlattenParamsWrapper(
2022-12-01T10:44:42.6613290Z     (_fpw_module): MyModule()
2022-12-01T10:44:42.6613516Z   )
2022-12-01T10:44:42.6613719Z )
2022-12-01T10:44:42.6614067Z ERROR: expected to be in states [<TrainingState_.IDLE: 1>] but current state is TrainingState_.FORWARD
2022-12-01T10:44:42.6614732Z ok (3.510s)
2022-12-01T10:44:42.6615174Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams)
2022-12-01T10:44:42.6615718Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21728
2022-12-01T10:44:42.6616239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21729
2022-12-01T10:44:42.6616839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6617291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6617867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6618326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6618905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6619345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6619914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6620360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6620811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6621303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6621965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6622640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6623161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6623634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6624484Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6625033Z   warnings.warn(
2022-12-01T10:44:42.6625870Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6626428Z   warnings.warn(
2022-12-01T10:44:42.6626729Z dist init r=0, world=2
2022-12-01T10:44:42.6626981Z dist init r=1, world=2
2022-12-01T10:44:42.6627220Z ok (3.412s)
2022-12-01T10:44:42.6627553Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams)
2022-12-01T10:44:42.6628102Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21807
2022-12-01T10:44:42.6628621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21808
2022-12-01T10:44:42.6629231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6629662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6630232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6630700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6631277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6631701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6632268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6632733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6633169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6633664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6634953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6635645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6636152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6636623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6637487Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6638034Z   warnings.warn(
2022-12-01T10:44:42.6638763Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6639310Z   warnings.warn(
2022-12-01T10:44:42.6639556Z dist init r=1, world=2
2022-12-01T10:44:42.6639807Z dist init r=0, world=2
2022-12-01T10:44:42.6640029Z ok (3.412s)
2022-12-01T10:44:42.6640423Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams)
2022-12-01T10:44:42.6640985Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21886
2022-12-01T10:44:42.6641489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21887
2022-12-01T10:44:42.6642099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6642549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6643125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6643644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6644235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6644754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6645310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6645778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6646231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6646726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6647365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6648057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6648580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6649054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6649903Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6650447Z   warnings.warn(
2022-12-01T10:44:42.6651200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6651737Z   warnings.warn(
2022-12-01T10:44:42.6651969Z dist init r=1, world=2
2022-12-01T10:44:42.6652219Z dist init r=0, world=2
2022-12-01T10:44:42.6652455Z ok (3.410s)
2022-12-01T10:44:42.6652804Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams)
2022-12-01T10:44:42.6653360Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21965
2022-12-01T10:44:42.6653883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21966
2022-12-01T10:44:42.6654877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6655346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6655925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6656394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6656959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6657410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6657986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6658433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6658886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6659383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6660036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6660706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6661334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6661826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6662769Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6663299Z   warnings.warn(
2022-12-01T10:44:42.6664052Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6664590Z   warnings.warn(
2022-12-01T10:44:42.6664841Z dist init r=0, world=2
2022-12-01T10:44:42.6665073Z dist init r=1, world=2
2022-12-01T10:44:42.6665310Z ok (3.410s)
2022-12-01T10:44:42.6665829Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22044
2022-12-01T10:44:42.6666423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22045
2022-12-01T10:44:42.6667026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6667472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6668047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6668495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6669071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6669517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6670067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6670536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6670992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6671486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6672121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6672804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6673325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6673804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6674652Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6675200Z   warnings.warn(
2022-12-01T10:44:42.6675946Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6676483Z   warnings.warn(
2022-12-01T10:44:42.6676712Z dist init r=0, world=2
2022-12-01T10:44:42.6676961Z dist init r=1, world=2
2022-12-01T10:44:42.6677202Z ok (3.310s)
2022-12-01T10:44:42.6677760Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22123
2022-12-01T10:44:42.6678377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22124
2022-12-01T10:44:42.6678989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6679495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6680055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6680545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6681125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6681549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6682118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6682585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6683037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6683519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6684175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6684860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6685386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6685836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6686697Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6687247Z   warnings.warn(
2022-12-01T10:44:42.6687999Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6688520Z   warnings.warn(
2022-12-01T10:44:42.6688766Z dist init r=0, world=2
2022-12-01T10:44:42.6689015Z dist init r=1, world=2
2022-12-01T10:44:42.6689237Z ok (3.411s)
2022-12-01T10:44:42.6689750Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22202
2022-12-01T10:44:42.6690347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22203
2022-12-01T10:44:42.6690986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6691415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6691989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6692456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6693014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6693458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6694026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6694844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6695387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6695935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6696681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6697351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6697876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6698352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6699438Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6700110Z   warnings.warn(
2022-12-01T10:44:42.6701065Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6701732Z   warnings.warn(
2022-12-01T10:44:42.6702490Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6703029Z   warnings.warn(
2022-12-01T10:44:42.6703762Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6704299Z   warnings.warn(
2022-12-01T10:44:42.6704551Z dist init r=0, world=2
2022-12-01T10:44:42.6704783Z dist init r=1, world=2
2022-12-01T10:44:42.6705019Z ok (3.510s)
2022-12-01T10:44:42.6705530Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22281
2022-12-01T10:44:42.6706132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22282
2022-12-01T10:44:42.6706726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6707174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6707755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6708245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6708809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6709256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6709846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6710295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6710745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6711236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6711892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6712625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6713221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6713690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6714775Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6715425Z   warnings.warn(
2022-12-01T10:44:42.6716396Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6717046Z   warnings.warn(
2022-12-01T10:44:42.6717802Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6718340Z   warnings.warn(
2022-12-01T10:44:42.6719068Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6719606Z   warnings.warn(
2022-12-01T10:44:42.6719852Z dist init r=1, world=2
2022-12-01T10:44:42.6720085Z dist init r=0, world=2
2022-12-01T10:44:42.6720320Z ok (3.511s)
2022-12-01T10:44:42.6720840Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22360
2022-12-01T10:44:42.6721451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22361
2022-12-01T10:44:42.6722045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6722493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6723065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6723520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6724117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6724566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6725135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6725583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6726035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6726532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6727182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6727871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6728393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6728923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6729785Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6730387Z   warnings.warn(
2022-12-01T10:44:42.6731141Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6731672Z   warnings.warn(
2022-12-01T10:44:42.6731901Z dist init r=0, world=2
2022-12-01T10:44:42.6732150Z dist init r=1, world=2
2022-12-01T10:44:42.6732385Z ok (3.411s)
2022-12-01T10:44:42.6732885Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22439
2022-12-01T10:44:42.6733491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22440
2022-12-01T10:44:42.6734104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6734916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6735491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6735957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6736531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6736973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6737531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6737992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6738447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6738931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6739588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6740276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6740796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6741251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6742111Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6742669Z   warnings.warn(
2022-12-01T10:44:42.6743420Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6743942Z   warnings.warn(
2022-12-01T10:44:42.6744187Z dist init r=0, world=2
2022-12-01T10:44:42.6744440Z dist init r=1, world=2
2022-12-01T10:44:42.6744660Z ok (3.411s)
2022-12-01T10:44:42.6745194Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22518
2022-12-01T10:44:42.6745810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22519
2022-12-01T10:44:42.6746508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6747012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6747595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6748066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6748645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6749068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6749631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6750092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6750530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6751022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6751677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6752364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6752867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6753339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6754198Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6754745Z   warnings.warn(
2022-12-01T10:44:42.6755475Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6756018Z   warnings.warn(
2022-12-01T10:44:42.6756266Z dist init r=0, world=2
2022-12-01T10:44:42.6756499Z dist init r=1, world=2
2022-12-01T10:44:42.6756740Z ok (3.511s)
2022-12-01T10:44:42.6757252Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22597
2022-12-01T10:44:42.6757853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22598
2022-12-01T10:44:42.6758444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6758894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6759468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6759925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6760498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6760943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6761513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6761960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6762411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6762966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6763636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6764366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6764884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6765354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6766221Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6766748Z   warnings.warn(
2022-12-01T10:44:42.6767506Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6768040Z   warnings.warn(
2022-12-01T10:44:42.6768275Z dist init r=1, world=2
2022-12-01T10:44:42.6768524Z dist init r=0, world=2
2022-12-01T10:44:42.6768761Z ok (3.511s)
2022-12-01T10:44:42.6769254Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22676
2022-12-01T10:44:42.6769859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22677
2022-12-01T10:44:42.6770466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6770915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6771475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6771945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6772522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6772969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6773521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6773988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6774879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6775420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6776096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6776785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6777312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6777767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6778634Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6779182Z   warnings.warn(
2022-12-01T10:44:42.6779928Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6780546Z   warnings.warn(
2022-12-01T10:44:42.6780810Z dist init r=0, world=2
2022-12-01T10:44:42.6781062Z dist init r=1, world=2
2022-12-01T10:44:42.6781282Z ok (3.410s)
2022-12-01T10:44:42.6781877Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22755
2022-12-01T10:44:42.6782473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22756
2022-12-01T10:44:42.6783087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6783515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6784091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6784556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6785139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6785566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6786138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6786600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6787033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6787556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6788213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6788900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6789410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6789878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6790747Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6791297Z   warnings.warn(
2022-12-01T10:44:42.6792028Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6792566Z   warnings.warn(
2022-12-01T10:44:42.6792813Z dist init r=1, world=2
2022-12-01T10:44:42.6793043Z dist init r=0, world=2
2022-12-01T10:44:42.6793282Z ok (3.410s)
2022-12-01T10:44:42.6793797Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22834
2022-12-01T10:44:42.6794400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22835
2022-12-01T10:44:42.6794993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6795445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6796071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6796547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6797108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6797609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6798198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6798702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6799157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6799652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6800312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6800978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6801502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6801979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6802845Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6803374Z   warnings.warn(
2022-12-01T10:44:42.6804123Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6804657Z   warnings.warn(
2022-12-01T10:44:42.6804903Z dist init r=1, world=2
2022-12-01T10:44:42.6805139Z dist init r=0, world=2
2022-12-01T10:44:42.6805380Z ok (3.410s)
2022-12-01T10:44:42.6805890Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22913
2022-12-01T10:44:42.6806474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22914
2022-12-01T10:44:42.6807088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6807538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6808112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6808561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6809137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6809578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6810133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6810600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6811052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6811551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6812190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6812876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6813395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6813866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6815306Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6816199Z   warnings.warn(
2022-12-01T10:44:42.6816973Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6817510Z   warnings.warn(
2022-12-01T10:44:42.6817740Z dist init r=1, world=2
2022-12-01T10:44:42.6817994Z dist init r=0, world=2
2022-12-01T10:44:42.6818233Z ok (3.410s)
2022-12-01T10:44:42.6818721Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22992
2022-12-01T10:44:42.6819316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22993
2022-12-01T10:44:42.6819931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6820361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6820942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6821405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6821977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6822399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6822969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6823429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6823885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6824360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6825018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6825709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6826209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6826679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6827536Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6828083Z   warnings.warn(
2022-12-01T10:44:42.6828811Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6829351Z   warnings.warn(
2022-12-01T10:44:42.6829597Z dist init r=1, world=2
2022-12-01T10:44:42.6829846Z dist init r=0, world=2
2022-12-01T10:44:42.6830063Z ok (3.410s)
2022-12-01T10:44:42.6830570Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23071
2022-12-01T10:44:42.6831168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23072
2022-12-01T10:44:42.6831754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6832267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6832856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6833380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6833942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6834386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6834957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6835418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6835851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6836346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6836998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6837664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6838183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6838653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6839514Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6840039Z   warnings.warn(
2022-12-01T10:44:42.6840849Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6841392Z   warnings.warn(
2022-12-01T10:44:42.6841650Z dist init r=0, world=2
2022-12-01T10:44:42.6841880Z dist init r=1, world=2
2022-12-01T10:44:42.6842117Z ok (3.510s)
2022-12-01T10:44:42.6842625Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23150
2022-12-01T10:44:42.6843206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23151
2022-12-01T10:44:42.6843815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6844263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6844841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6845295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6845870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6846319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6846871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6847334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6847787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6848279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6848916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6849661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6850246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6850717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6851570Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6852115Z   warnings.warn(
2022-12-01T10:44:42.6852858Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6853386Z   warnings.warn(
2022-12-01T10:44:42.6853618Z dist init r=0, world=2
2022-12-01T10:44:42.6853868Z dist init r=1, world=2
2022-12-01T10:44:42.6854106Z ok (3.410s)
2022-12-01T10:44:42.6855005Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23229
2022-12-01T10:44:42.6855946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23230
2022-12-01T10:44:42.6856567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6857019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6857575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6858041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6858621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6859044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6859621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6860083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6860535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6861012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6861668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6862354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6862881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6863339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6864203Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6864749Z   warnings.warn(
2022-12-01T10:44:42.6865502Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6866020Z   warnings.warn(
2022-12-01T10:44:42.6866269Z dist init r=1, world=2
2022-12-01T10:44:42.6866522Z dist init r=0, world=2
2022-12-01T10:44:42.6866739Z ok (3.410s)
2022-12-01T10:44:42.6867148Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams)
2022-12-01T10:44:42.6867679Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23308
2022-12-01T10:44:42.6868247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23309
2022-12-01T10:44:42.6868861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6869311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6869884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6870331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6870908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6871353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6871904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6872371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6872824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6873321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6873962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6874651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6875173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6875648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6875986Z dist init r=0, world=2
2022-12-01T10:44:42.6876236Z dist init r=1, world=2
2022-12-01T10:44:42.6876477Z ok (3.409s)
2022-12-01T10:44:42.6876997Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23387
2022-12-01T10:44:42.6877633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23388
2022-12-01T10:44:42.6878245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6878695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6879249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6879717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6880291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6880720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6881291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6881753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6882202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6882678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6883330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6884077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6884612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6885157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6886373Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6887125Z   warnings.warn(
2022-12-01T10:44:42.6888236Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6888981Z   warnings.warn(
2022-12-01T10:44:42.6889718Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6890255Z   warnings.warn(
2022-12-01T10:44:42.6891000Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6891535Z   warnings.warn(
2022-12-01T10:44:42.6892282Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6892830Z   warnings.warn(
2022-12-01T10:44:42.6893584Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6894124Z   warnings.warn(
2022-12-01T10:44:42.6894519Z dist init r=1, world=2
2022-12-01T10:44:42.6894972Z dist init r=0, world=2
2022-12-01T10:44:42.6895411Z ok (3.912s)
2022-12-01T10:44:42.6896065Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23470
2022-12-01T10:44:42.6896705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23471
2022-12-01T10:44:42.6897328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6897788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6898340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6898808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6899392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6899817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6900388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6900852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6901409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6901907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6902635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6903322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6903841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6904295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6905504Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6906248Z   warnings.warn(
2022-12-01T10:44:42.6907354Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6908085Z   warnings.warn(
2022-12-01T10:44:42.6908824Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6909361Z   warnings.warn(
2022-12-01T10:44:42.6910102Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6910646Z   warnings.warn(
2022-12-01T10:44:42.6911385Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6911934Z   warnings.warn(
2022-12-01T10:44:42.6912684Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6913219Z   warnings.warn(
2022-12-01T10:44:42.6913448Z dist init r=0, world=2
2022-12-01T10:44:42.6913696Z dist init r=1, world=2
2022-12-01T10:44:42.6913932Z ok (3.912s)
2022-12-01T10:44:42.6914455Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23553
2022-12-01T10:44:42.6915080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23554
2022-12-01T10:44:42.6915688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6916139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6916693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6917163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6917797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6918236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6918870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6919328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6919772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6920271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6920930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6921602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6922120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6922589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6923806Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6924537Z   warnings.warn(
2022-12-01T10:44:42.6925636Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6926376Z   warnings.warn(
2022-12-01T10:44:42.6927128Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6927657Z   warnings.warn(
2022-12-01T10:44:42.6928386Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6928915Z   warnings.warn(
2022-12-01T10:44:42.6929684Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6930218Z   warnings.warn(
2022-12-01T10:44:42.6930951Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6931491Z   warnings.warn(
2022-12-01T10:44:42.6932457Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6933114Z   warnings.warn(
2022-12-01T10:44:42.6934116Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6935469Z   warnings.warn(
2022-12-01T10:44:42.6935888Z dist init r=1, world=2
2022-12-01T10:44:42.6936141Z dist init r=0, world=2
2022-12-01T10:44:42.6936360Z ok (4.012s)
2022-12-01T10:44:42.6936899Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23636
2022-12-01T10:44:42.6937520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23637
2022-12-01T10:44:42.6938122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6938574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6939145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6939621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6940182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6940629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6941200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6941661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6942095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6942593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6943248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6943920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6944439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6944909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6946128Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6946871Z   warnings.warn(
2022-12-01T10:44:42.6947960Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6948699Z   warnings.warn(
2022-12-01T10:44:42.6949455Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6949991Z   warnings.warn(
2022-12-01T10:44:42.6950813Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6951366Z   warnings.warn(
2022-12-01T10:44:42.6952196Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6952741Z   warnings.warn(
2022-12-01T10:44:42.6953471Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6954002Z   warnings.warn(
2022-12-01T10:44:42.6954971Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6955630Z   warnings.warn(
2022-12-01T10:44:42.6956583Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.6957222Z   warnings.warn(
2022-12-01T10:44:42.6957467Z dist init r=0, world=2
2022-12-01T10:44:42.6957715Z dist init r=1, world=2
2022-12-01T10:44:42.6957934Z ok (3.912s)
2022-12-01T10:44:42.6958473Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23719
2022-12-01T10:44:42.6959104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23720
2022-12-01T10:44:42.6959716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6960147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6960714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6961184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6961759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6962183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6962749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6963236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6963673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6964167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6964823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6965513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6966016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6966485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6967757Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6968556Z   warnings.warn(
2022-12-01T10:44:42.6969661Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6970376Z   warnings.warn(
2022-12-01T10:44:42.6971131Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6971666Z   warnings.warn(
2022-12-01T10:44:42.6972402Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6972918Z   warnings.warn(
2022-12-01T10:44:42.6973674Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6974216Z   warnings.warn(
2022-12-01T10:44:42.6975831Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6976369Z   warnings.warn(
2022-12-01T10:44:42.6976612Z dist init r=0, world=2
2022-12-01T10:44:42.6976869Z dist init r=1, world=2
2022-12-01T10:44:42.6977088Z ok (4.012s)
2022-12-01T10:44:42.6977623Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23802
2022-12-01T10:44:42.6978242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23803
2022-12-01T10:44:42.6978856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6979285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6979857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6980325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6980904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6981335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6981905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6982368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6982802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.6983296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.6983947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6984728Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.6985250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.6985793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.6987011Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6987758Z   warnings.warn(
2022-12-01T10:44:42.6988859Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.6989587Z   warnings.warn(
2022-12-01T10:44:42.6990346Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6990876Z   warnings.warn(
2022-12-01T10:44:42.6991616Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.6992135Z   warnings.warn(
2022-12-01T10:44:42.6992893Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6993441Z   warnings.warn(
2022-12-01T10:44:42.6994193Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.6994708Z   warnings.warn(
2022-12-01T10:44:42.6994948Z dist init r=1, world=2
2022-12-01T10:44:42.6995197Z dist init r=0, world=2
2022-12-01T10:44:42.6995417Z ok (3.912s)
2022-12-01T10:44:42.6995988Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23885
2022-12-01T10:44:42.6996620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23886
2022-12-01T10:44:42.6997228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6997661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.6998231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.6998696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.6999253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.6999694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7000261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7000783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7001231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7001783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7002441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7003129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7003633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7004102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7005320Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7006073Z   warnings.warn(
2022-12-01T10:44:42.7007162Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7007905Z   warnings.warn(
2022-12-01T10:44:42.7008663Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7009197Z   warnings.warn(
2022-12-01T10:44:42.7009942Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7010464Z   warnings.warn(
2022-12-01T10:44:42.7011221Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.7011765Z   warnings.warn(
2022-12-01T10:44:42.7012519Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.7013035Z   warnings.warn(
2022-12-01T10:44:42.7013281Z dist init r=0, world=2
2022-12-01T10:44:42.7013529Z dist init r=1, world=2
2022-12-01T10:44:42.7013752Z ok (3.912s)
2022-12-01T10:44:42.7014282Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23968
2022-12-01T10:44:42.7015628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23969
2022-12-01T10:44:42.7016286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7016710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7017280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7017847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7018425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7018941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7019515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7019978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7020411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7020903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7021557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7022248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7022751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7023223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7024434Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7025177Z   warnings.warn(
2022-12-01T10:44:42.7026265Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7027013Z   warnings.warn(
2022-12-01T10:44:42.7027766Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7028301Z   warnings.warn(
2022-12-01T10:44:42.7029038Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7029554Z   warnings.warn(
2022-12-01T10:44:42.7030312Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.7030851Z   warnings.warn(
2022-12-01T10:44:42.7031596Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:44:42.7032111Z   warnings.warn(
2022-12-01T10:44:42.7032354Z dist init r=1, world=2
2022-12-01T10:44:42.7032603Z dist init r=0, world=2
2022-12-01T10:44:42.7032823Z ok (3.912s)
2022-12-01T10:44:42.7033252Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24051
2022-12-01T10:44:42.7033779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24052
2022-12-01T10:44:42.7034440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7034878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7035504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7035972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7036529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7036976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7037541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7038002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7038437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7038935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7039593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7040267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7040784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7041255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7042118Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7042645Z   warnings.warn(
2022-12-01T10:44:42.7043396Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7043936Z   warnings.warn(
2022-12-01T10:44:42.7044181Z dist init r=0, world=2
2022-12-01T10:44:42.7044412Z dist init r=1, world=2
2022-12-01T10:44:42.7044649Z ok (3.511s)
2022-12-01T10:44:42.7045158Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24130
2022-12-01T10:44:42.7045739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24131
2022-12-01T10:44:42.7046345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7046790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7047366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7048011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7048796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7049243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7049814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7050261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7050712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7051209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7051914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7052619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7053226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7053696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7055631Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7056417Z   warnings.warn(
2022-12-01T10:44:42.7057539Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7058282Z   warnings.warn(
2022-12-01T10:44:42.7059031Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7059547Z   warnings.warn(
2022-12-01T10:44:42.7060292Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7060827Z   warnings.warn(
2022-12-01T10:44:42.7061074Z dist init r=0, world=2
2022-12-01T10:44:42.7061310Z dist init r=1, world=2
2022-12-01T10:44:42.7061546Z ok (3.410s)
2022-12-01T10:44:42.7062060Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24209
2022-12-01T10:44:42.7062645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24210
2022-12-01T10:44:42.7063254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7063702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7064275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7064729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7065305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7065753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7066318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7066765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7067219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7067713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7068349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7069123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7069660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7070197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7071399Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7072150Z   warnings.warn(
2022-12-01T10:44:42.7073254Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7073997Z   warnings.warn(
2022-12-01T10:44:42.7074750Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7075270Z   warnings.warn(
2022-12-01T10:44:42.7076010Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7076539Z   warnings.warn(
2022-12-01T10:44:42.7076785Z dist init r=1, world=2
2022-12-01T10:44:42.7077020Z dist init r=0, world=2
2022-12-01T10:44:42.7077256Z ok (3.510s)
2022-12-01T10:44:42.7077768Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24288
2022-12-01T10:44:42.7078358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24289
2022-12-01T10:44:42.7078964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7079413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7079984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7080432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7081009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7081455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7082007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7082479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7082931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7083422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7084090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7084776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7085354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7085838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7087101Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7087850Z   warnings.warn(
2022-12-01T10:44:42.7088957Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7089688Z   warnings.warn(
2022-12-01T10:44:42.7090446Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7090968Z   warnings.warn(
2022-12-01T10:44:42.7091708Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7092245Z   warnings.warn(
2022-12-01T10:44:42.7092492Z dist init r=0, world=2
2022-12-01T10:44:42.7092721Z dist init r=1, world=2
2022-12-01T10:44:42.7092956Z ok (3.511s)
2022-12-01T10:44:42.7093467Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24367
2022-12-01T10:44:42.7094052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24368
2022-12-01T10:44:42.7095127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7095866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7096487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7096939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7097517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7097959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7098516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7098985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7099442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7099958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7100593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7101278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7101793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7102261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7103554Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7104368Z   warnings.warn(
2022-12-01T10:44:42.7105475Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7106219Z   warnings.warn(
2022-12-01T10:44:42.7106973Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7107500Z   warnings.warn(
2022-12-01T10:44:42.7108240Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7108776Z   warnings.warn(
2022-12-01T10:44:42.7109023Z dist init r=0, world=2
2022-12-01T10:44:42.7109253Z dist init r=1, world=2
2022-12-01T10:44:42.7109489Z ok (3.411s)
2022-12-01T10:44:42.7110001Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24446
2022-12-01T10:44:42.7110586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24447
2022-12-01T10:44:42.7111198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7111654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7112225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7112399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7112768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7112939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7113338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7113537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7113783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7114022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7114423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7114802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7115033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7115282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7116322Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7116490Z   warnings.warn(
2022-12-01T10:44:42.7117462Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7117571Z   warnings.warn(
2022-12-01T10:44:42.7118193Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7118300Z   warnings.warn(
2022-12-01T10:44:42.7118904Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7119015Z   warnings.warn(
2022-12-01T10:44:42.7119126Z dist init r=0, world=2
2022-12-01T10:44:42.7119217Z dist init r=1, world=2
2022-12-01T10:44:42.7119317Z ok (3.410s)
2022-12-01T10:44:42.7119702Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24525
2022-12-01T10:44:42.7119923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24526
2022-12-01T10:44:42.7120297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7120473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7120851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7121047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7121396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7121570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7121949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7122143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7122390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7122632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7123035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7123437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7123669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7123881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7124919Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7125042Z   warnings.warn(
2022-12-01T10:44:42.7126013Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7126176Z   warnings.warn(
2022-12-01T10:44:42.7126792Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7126899Z   warnings.warn(
2022-12-01T10:44:42.7127505Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7127617Z   warnings.warn(
2022-12-01T10:44:42.7127728Z dist init r=0, world=2
2022-12-01T10:44:42.7127839Z dist init r=1, world=2
2022-12-01T10:44:42.7127921Z ok (3.511s)
2022-12-01T10:44:42.7128306Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24604
2022-12-01T10:44:42.7128526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24605
2022-12-01T10:44:42.7128898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7129073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7129457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7129653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7130025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7130182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7130560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7130753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7131015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7131255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7131659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7132057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7132290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7132517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7133489Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7133584Z   warnings.warn(
2022-12-01T10:44:42.7135188Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7135498Z   warnings.warn(
2022-12-01T10:44:42.7136238Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7136347Z   warnings.warn(
2022-12-01T10:44:42.7136958Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7137066Z   warnings.warn(
2022-12-01T10:44:42.7137176Z dist init r=0, world=2
2022-12-01T10:44:42.7137284Z dist init r=1, world=2
2022-12-01T10:44:42.7137386Z ok (3.410s)
2022-12-01T10:44:42.7137752Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24683
2022-12-01T10:44:42.7137973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24684
2022-12-01T10:44:42.7138364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7138541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7138923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7139120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7139488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7139668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7140047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7140219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7140466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7140778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7141186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7141585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7141815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7142049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7143022Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7143135Z   warnings.warn(
2022-12-01T10:44:42.7144161Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7144326Z   warnings.warn(
2022-12-01T10:44:42.7144934Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7145044Z   warnings.warn(
2022-12-01T10:44:42.7145650Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7145757Z   warnings.warn(
2022-12-01T10:44:42.7145869Z dist init r=1, world=2
2022-12-01T10:44:42.7145977Z dist init r=0, world=2
2022-12-01T10:44:42.7146080Z ok (3.410s)
2022-12-01T10:44:42.7146432Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24762
2022-12-01T10:44:42.7146639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24763
2022-12-01T10:44:42.7147011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7147185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7147563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7147755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7148122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7148302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7148682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7148875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7149103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7149342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7149741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7150136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7150366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7150598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7151213Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7151328Z   warnings.warn(
2022-12-01T10:44:42.7151928Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7152021Z   warnings.warn(
2022-12-01T10:44:42.7152130Z dist init r=0, world=2
2022-12-01T10:44:42.7152240Z dist init r=1, world=2
2022-12-01T10:44:42.7152339Z ok (3.410s)
2022-12-01T10:44:42.7152739Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24841
2022-12-01T10:44:42.7152966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24842
2022-12-01T10:44:42.7153340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7153572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7153934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7154125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7154491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7154665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7155044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7155239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7155485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7155725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7156146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7156531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7156759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7156985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7157600Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7157713Z   warnings.warn(
2022-12-01T10:44:42.7158321Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7158430Z   warnings.warn(
2022-12-01T10:44:42.7158542Z dist init r=1, world=2
2022-12-01T10:44:42.7158650Z dist init r=0, world=2
2022-12-01T10:44:42.7158732Z ok (3.410s)
2022-12-01T10:44:42.7159189Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24920
2022-12-01T10:44:42.7159407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24921
2022-12-01T10:44:42.7159781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7159960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7160339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7160531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7160898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7161052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7161427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7161636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7161958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7162208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7162658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7163053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7163283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7163510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7164112Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7164229Z   warnings.warn(
2022-12-01T10:44:42.7164838Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7164950Z   warnings.warn(
2022-12-01T10:44:42.7165060Z dist init r=0, world=2
2022-12-01T10:44:42.7165168Z dist init r=1, world=2
2022-12-01T10:44:42.7165268Z ok (3.410s)
2022-12-01T10:44:42.7165722Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24999
2022-12-01T10:44:42.7165941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25000
2022-12-01T10:44:42.7166296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7166474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7166848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7167044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7167412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7167585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7167965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7168155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7168399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7168626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7169027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7169424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7169655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7169878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7170500Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7170610Z   warnings.warn(
2022-12-01T10:44:42.7171271Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7171431Z   warnings.warn(
2022-12-01T10:44:42.7171524Z dist init r=1, world=2
2022-12-01T10:44:42.7171635Z dist init r=0, world=2
2022-12-01T10:44:42.7171734Z ok (3.410s)
2022-12-01T10:44:42.7172186Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25078
2022-12-01T10:44:42.7172408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25079
2022-12-01T10:44:42.7172784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7172959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7173339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7173518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7173885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7174057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7174786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7175151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7175617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7175932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7176352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7176751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7176968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7177200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7177820Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7177933Z   warnings.warn(
2022-12-01T10:44:42.7178543Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7178651Z   warnings.warn(
2022-12-01T10:44:42.7178761Z dist init r=0, world=2
2022-12-01T10:44:42.7178873Z dist init r=1, world=2
2022-12-01T10:44:42.7178972Z ok (3.510s)
2022-12-01T10:44:42.7179406Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25157
2022-12-01T10:44:42.7179628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25158
2022-12-01T10:44:42.7179999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7180175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7180648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7180852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7181223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7181466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7181846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7182019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7182267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7182503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7182904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7183303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7183538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7183768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7184386Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7184497Z   warnings.warn(
2022-12-01T10:44:42.7185086Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7185195Z   warnings.warn(
2022-12-01T10:44:42.7185309Z dist init r=1, world=2
2022-12-01T10:44:42.7185419Z dist init r=0, world=2
2022-12-01T10:44:42.7185519Z ok (3.410s)
2022-12-01T10:44:42.7185969Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25236
2022-12-01T10:44:42.7186194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25237
2022-12-01T10:44:42.7186567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7186724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7187103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7187294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7187662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7187837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7188220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7188410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7188655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7188891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7189297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7189761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7189998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7190227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7190904Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7191017Z   warnings.warn(
2022-12-01T10:44:42.7191621Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7191727Z   warnings.warn(
2022-12-01T10:44:42.7191838Z dist init r=1, world=2
2022-12-01T10:44:42.7191932Z dist init r=0, world=2
2022-12-01T10:44:42.7192032Z ok (3.410s)
2022-12-01T10:44:42.7192485Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25315
2022-12-01T10:44:42.7192707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25316
2022-12-01T10:44:42.7193077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7193255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7193634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7193827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7194191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7194352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7194728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7194920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7195166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7195566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7195808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7196247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7196479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7196691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7197309Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7197428Z   warnings.warn(
2022-12-01T10:44:42.7198039Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7198147Z   warnings.warn(
2022-12-01T10:44:42.7198257Z dist init r=1, world=2
2022-12-01T10:44:42.7198370Z dist init r=0, world=2
2022-12-01T10:44:42.7198484Z ok (3.410s)
2022-12-01T10:44:42.7198990Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25394
2022-12-01T10:44:42.7199244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25395
2022-12-01T10:44:42.7199618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7199793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7200172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7200363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7200730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7200904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7201286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7201479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7201712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7201950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7202351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7202750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7202981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7203207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7203830Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7203945Z   warnings.warn(
2022-12-01T10:44:42.7204552Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7204643Z   warnings.warn(
2022-12-01T10:44:42.7204754Z dist init r=0, world=2
2022-12-01T10:44:42.7204861Z dist init r=1, world=2
2022-12-01T10:44:42.7204960Z ok (3.410s)
2022-12-01T10:44:42.7205411Z   test_summon_full_param_writeback_writeback_False_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25473
2022-12-01T10:44:42.7205631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25474
2022-12-01T10:44:42.7206001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7206182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7206541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7206734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7207100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7207273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7207648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7207896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7208151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7208437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7208839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7209215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7209444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7209670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7210291Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7210405Z   warnings.warn(
2022-12-01T10:44:42.7211010Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7211117Z   warnings.warn(
2022-12-01T10:44:42.7211224Z dist init r=0, world=2
2022-12-01T10:44:42.7231381Z dist init r=1, world=2
2022-12-01T10:44:42.7231542Z ok (3.510s)
2022-12-01T10:44:42.7232026Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25552
2022-12-01T10:44:42.7232258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25553
2022-12-01T10:44:42.7232660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7232844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7233225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7233419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7233784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7233959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7234334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7234523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7234778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7235007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7235415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7235816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7236046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7236273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7237043Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7237175Z   warnings.warn(
2022-12-01T10:44:42.7237808Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7237998Z   warnings.warn(
2022-12-01T10:44:42.7238093Z dist init r=1, world=2
2022-12-01T10:44:42.7238202Z dist init r=0, world=2
2022-12-01T10:44:42.7238303Z ok (3.409s)
2022-12-01T10:44:42.7238753Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25631
2022-12-01T10:44:42.7238976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25632
2022-12-01T10:44:42.7239355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7239531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7239911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7240106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7240450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7240623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7240996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7241183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7241429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7241678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7242076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7242477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7242690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7242916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7243537Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7243651Z   warnings.warn(
2022-12-01T10:44:42.7244267Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7244380Z   warnings.warn(
2022-12-01T10:44:42.7244492Z dist init r=1, world=2
2022-12-01T10:44:42.7244599Z dist init r=0, world=2
2022-12-01T10:44:42.7244697Z ok (3.410s)
2022-12-01T10:44:42.7245128Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25710
2022-12-01T10:44:42.7245351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25711
2022-12-01T10:44:42.7245722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7245899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7246342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7246585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7246953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7247127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7247499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7247669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7247915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7248159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7248561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7248958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7249189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7249417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7250035Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7250148Z   warnings.warn(
2022-12-01T10:44:42.7250745Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7250857Z   warnings.warn(
2022-12-01T10:44:42.7250967Z dist init r=1, world=2
2022-12-01T10:44:42.7251081Z dist init r=0, world=2
2022-12-01T10:44:42.7251180Z ok (3.410s)
2022-12-01T10:44:42.7251628Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=False)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25789
2022-12-01T10:44:42.7251846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25790
2022-12-01T10:44:42.7252217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7252374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7252754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7252945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7253310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7253485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7253857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7254045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7254289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7254975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7255745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7256353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7256677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7256931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7257580Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7257692Z   warnings.warn(
2022-12-01T10:44:42.7258309Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7258419Z   warnings.warn(
2022-12-01T10:44:42.7258534Z dist init r=1, world=2
2022-12-01T10:44:42.7258627Z dist init r=0, world=2
2022-12-01T10:44:42.7258727Z ok (3.410s)
2022-12-01T10:44:42.7259176Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25868
2022-12-01T10:44:42.7259401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25869
2022-12-01T10:44:42.7259773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7259949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7260328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7260519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7260891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7261050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7261429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7261616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7261860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7262101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7262502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7262903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7263133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7263363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7263966Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7264078Z   warnings.warn(
2022-12-01T10:44:42.7264694Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7264806Z   warnings.warn(
2022-12-01T10:44:42.7264917Z dist init r=1, world=2
2022-12-01T10:44:42.7265026Z dist init r=0, world=2
2022-12-01T10:44:42.7265125Z ok (3.410s)
2022-12-01T10:44:42.7265647Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_False_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25947
2022-12-01T10:44:42.7265903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25948
2022-12-01T10:44:42.7266278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7266455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7266834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7267025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7267415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7267593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7267985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7268176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7268406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7268649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7269047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7269439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7269669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7269900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7270519Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7270636Z   warnings.warn(
2022-12-01T10:44:42.7271273Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7271381Z   warnings.warn(
2022-12-01T10:44:42.7271476Z dist init r=1, world=2
2022-12-01T10:44:42.7271584Z dist init r=0, world=2
2022-12-01T10:44:42.7271683Z ok (3.510s)
2022-12-01T10:44:42.7272132Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26026
2022-12-01T10:44:42.7272356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26027
2022-12-01T10:44:42.7272729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7272904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7273283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7273457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7273822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7273995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7274428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7274639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7274948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7275205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7275627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7276026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7276269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7276512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7277169Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7277286Z   warnings.warn(
2022-12-01T10:44:42.7277941Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7278055Z   warnings.warn(
2022-12-01T10:44:42.7278170Z dist init r=0, world=2
2022-12-01T10:44:42.7278282Z dist init r=1, world=2
2022-12-01T10:44:42.7278366Z ok (3.410s)
2022-12-01T10:44:42.7278846Z   test_summon_full_param_writeback_writeback_True_cpu_offload_CPUOffload(offload_params=True)_mixed_precision_True_modify_outer_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26105
2022-12-01T10:44:42.7279075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26106
2022-12-01T10:44:42.7279469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7279653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7280057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7280255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7280645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7280827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7281209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7281410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7281666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7281924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7282372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7282783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7283018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7283244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7283924Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7284028Z   warnings.warn(
2022-12-01T10:44:42.7284738Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7284851Z   warnings.warn(
2022-12-01T10:44:42.7284963Z dist init r=0, world=2
2022-12-01T10:44:42.7285072Z dist init r=1, world=2
2022-12-01T10:44:42.7285172Z ok (3.411s)
2022-12-01T10:44:42.7285543Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26184
2022-12-01T10:44:42.7285762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26185
2022-12-01T10:44:42.7286120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7286299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7286682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7286875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7287241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7287416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7287790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7287979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7288228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7288454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7288859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7289254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7289483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7289711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7290328Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7290440Z   warnings.warn(
2022-12-01T10:44:42.7291060Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7291173Z   warnings.warn(
2022-12-01T10:44:42.7291266Z dist init r=1, world=2
2022-12-01T10:44:42.7291375Z dist init r=0, world=2
2022-12-01T10:44:42.7291474Z ok (3.411s)
2022-12-01T10:44:42.7291838Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26263
2022-12-01T10:44:42.7292058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26264
2022-12-01T10:44:42.7292431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7292607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7293043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7293268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7293636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7293808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7294183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7294748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7295230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7295693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7296168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7296569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7296785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7297014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7297852Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.7297964Z   warnings.warn(
2022-12-01T10:44:42.7298791Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:2950: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2022-12-01T10:44:42.7298907Z   warnings.warn(
2022-12-01T10:44:42.7299528Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7299637Z   warnings.warn(
2022-12-01T10:44:42.7300252Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7300361Z   warnings.warn(
2022-12-01T10:44:42.7300478Z dist init r=1, world=2
2022-12-01T10:44:42.7300571Z dist init r=0, world=2
2022-12-01T10:44:42.7300674Z ok (3.410s)
2022-12-01T10:44:42.7301042Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26342
2022-12-01T10:44:42.7301266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26343
2022-12-01T10:44:42.7301640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7301817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7302195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7302385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7302833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7303022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7303467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7303656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7303900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7304146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7304546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7304939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7305172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7305383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7306013Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7306127Z   warnings.warn(
2022-12-01T10:44:42.7306742Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7306851Z   warnings.warn(
2022-12-01T10:44:42.7306963Z dist init r=1, world=2
2022-12-01T10:44:42.7307072Z dist init r=0, world=2
2022-12-01T10:44:42.7307172Z ok (3.510s)
2022-12-01T10:44:42.7307520Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26421
2022-12-01T10:44:42.7307745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26422
2022-12-01T10:44:42.7308118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7308294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7308662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7308836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7309211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7309401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7309779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7309951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7310200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7310442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7310841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7311233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7311464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7311692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7312368Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7312533Z   warnings.warn(
2022-12-01T10:44:42.7313130Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7313242Z   warnings.warn(
2022-12-01T10:44:42.7313353Z dist init r=1, world=2
2022-12-01T10:44:42.7313462Z dist init r=0, world=2
2022-12-01T10:44:42.7313562Z ok (3.510s)
2022-12-01T10:44:42.7313940Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26500
2022-12-01T10:44:42.7314162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26501
2022-12-01T10:44:42.7314532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7314711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7315071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7315260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7315622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7315798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7316169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7316359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7316603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7316850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7317231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7317630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7317856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7318076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7319048Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7319162Z   warnings.warn(
2022-12-01T10:44:42.7320124Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7320234Z   warnings.warn(
2022-12-01T10:44:42.7320902Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7321011Z   warnings.warn(
2022-12-01T10:44:42.7321618Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7321778Z   warnings.warn(
2022-12-01T10:44:42.7321873Z dist init r=1, world=2
2022-12-01T10:44:42.7321982Z dist init r=0, world=2
2022-12-01T10:44:42.7322081Z ok (3.911s)
2022-12-01T10:44:42.7322457Z   test_summon_full_params_respects_reshard_after_forward_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26579
2022-12-01T10:44:42.7322677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26580
2022-12-01T10:44:42.7323054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7323230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7323614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7323795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7324162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7324336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7324706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7324894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7325142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7325388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7325818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7326218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7326430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7326660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7327632Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7327746Z   warnings.warn(
2022-12-01T10:44:42.7328695Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7328806Z   warnings.warn(
2022-12-01T10:44:42.7329423Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7329530Z   warnings.warn(
2022-12-01T10:44:42.7330217Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7330375Z   warnings.warn(
2022-12-01T10:44:42.7330487Z dist init r=1, world=2
2022-12-01T10:44:42.7330578Z dist init r=0, world=2
2022-12-01T10:44:42.7330677Z ok (3.911s)
2022-12-01T10:44:42.7330987Z   test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26658
2022-12-01T10:44:42.7331205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26659
2022-12-01T10:44:42.7331579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7331752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7332134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7332327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7332674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7332852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7333224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7333410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7333655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7333900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:44:42.7334301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7335338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:44:42.7335759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:44:42.7335979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7336962Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7337075Z   warnings.warn(
2022-12-01T10:44:42.7338040Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:44:42.7338154Z   warnings.warn(
2022-12-01T10:44:42.7338769Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7338878Z   warnings.warn(
2022-12-01T10:44:42.7339488Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:44:42.7339596Z   warnings.warn(
2022-12-01T10:44:42.7339805Z dist init r=0, world=2
2022-12-01T10:44:42.7339930Z dist init r=1, world=2
2022-12-01T10:44:42.7340013Z ok (3.410s)
2022-12-01T10:44:42.7340494Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26737
2022-12-01T10:44:42.7340870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7341048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7341426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7341619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7341864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7342265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7342477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7342594Z dist init r=0, world=1
2022-12-01T10:44:42.7342694Z ok (3.308s)
2022-12-01T10:44:42.7343101Z   test_summon_full_param_writeback_writeback_False_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26777
2022-12-01T10:44:42.7343474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7343651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7344026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7344220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7344464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7344853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7345082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7345193Z dist init r=0, world=1
2022-12-01T10:44:42.7345292Z ok (3.208s)
2022-12-01T10:44:42.7345699Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26817
2022-12-01T10:44:42.7346070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7346248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7346626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7346803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7347046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7347444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7347673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7347783Z dist init r=0, world=1
2022-12-01T10:44:42.7347882Z ok (3.308s)
2022-12-01T10:44:42.7348286Z   test_summon_full_param_writeback_writeback_False_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26857
2022-12-01T10:44:42.7348710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7348892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7349305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7349496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7349739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7350138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7350365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7350477Z dist init r=0, world=1
2022-12-01T10:44:42.7350576Z ok (3.308s)
2022-12-01T10:44:42.7350986Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26897
2022-12-01T10:44:42.7351360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7351518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7351895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7352086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7352332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7352731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7352963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7353075Z dist init r=0, world=1
2022-12-01T10:44:42.7353174Z ok (3.308s)
2022-12-01T10:44:42.7353560Z   test_summon_full_param_writeback_writeback_True_modify_outer_False_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26937
2022-12-01T10:44:42.7353938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7354111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7354487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7354676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7354920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7355316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7355544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7355658Z dist init r=0, world=1
2022-12-01T10:44:42.7355740Z ok (3.208s)
2022-12-01T10:44:42.7356141Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_False (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26977
2022-12-01T10:44:42.7356509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7356686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7357059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7357311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7357565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7358051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7358260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7358377Z dist init r=0, world=1
2022-12-01T10:44:42.7358475Z ok (3.208s)
2022-12-01T10:44:42.7358878Z   test_summon_full_param_writeback_writeback_True_modify_outer_True_mixed_precision_True (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27017
2022-12-01T10:44:42.7359247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:44:42.7359421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:44:42.7359803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:44:42.7359997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:44:42.7360240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:44:42.7360619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:44:42.7360848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:44:42.7360958Z dist init r=0, world=1
2022-12-01T10:44:42.7361057Z ok (3.308s)
2022-12-01T10:44:42.7361081Z 
2022-12-01T10:44:42.7361355Z ----------------------------------------------------------------------
2022-12-01T10:44:42.7361470Z Ran 73 tests in 256.509s
2022-12-01T10:44:42.7361490Z 
2022-12-01T10:44:42.7361582Z OK
2022-12-01T10:44:42.7361601Z 
2022-12-01T10:44:42.7361727Z Generating XML reports...
2022-12-01T10:44:42.7362189Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221201104025.xml
2022-12-01T10:44:42.7362696Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221201104025.xml
2022-12-01T10:44:42.7362716Z 
2022-12-01T10:44:42.7363179Z ##[endgroup]
2022-12-01T10:44:42.7363695Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_eptkzh4h)
2022-12-01T10:44:42.7363716Z 
2022-12-01T10:44:42.7363990Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2022-12-01 10:44:42.654321]
2022-12-01T10:44:42.7364468Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:44:42.654597]
2022-12-01T10:46:29.3723842Z 
2022-12-01T10:46:29.3724426Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks
2022-12-01T10:46:29.3728314Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_n24fq9c8)
2022-12-01T10:46:29.3728722Z 
2022-12-01T10:46:29.3728859Z Running tests...
2022-12-01T10:46:29.3729378Z ----------------------------------------------------------------------
2022-12-01T10:46:29.3729937Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2022-12-01T10:46:29.3733535Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:46:29.3734192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27092
2022-12-01T10:46:29.3735231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27093
2022-12-01T10:46:29.3735946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3736523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3737284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3737792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3738446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3738939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3739576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3740060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3740536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3741025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3741750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3742372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3742907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3743380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3744249Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3744863Z   warnings.warn(
2022-12-01T10:46:29.3745647Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3746194Z   warnings.warn(
2022-12-01T10:46:29.3746948Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3747485Z   warnings.warn(
2022-12-01T10:46:29.3748252Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3748806Z   warnings.warn(
2022-12-01T10:46:29.3749026Z dist init r=0, world=2
2022-12-01T10:46:29.3749465Z dist init r=1, world=2
2022-12-01T10:46:29.3749694Z ok (5.494s)
2022-12-01T10:46:29.3750209Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27175
2022-12-01T10:46:29.3750817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27176
2022-12-01T10:46:29.3751458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3751918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3752494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3752940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3753672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3754112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3754747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3755226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3755703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3756191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3756848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3757695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3758253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3758710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3759601Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3760144Z   warnings.warn(
2022-12-01T10:46:29.3760895Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3761442Z   warnings.warn(
2022-12-01T10:46:29.3761673Z dist init r=0, world=2
2022-12-01T10:46:29.3761924Z dist init r=1, world=2
2022-12-01T10:46:29.3762163Z ok (3.911s)
2022-12-01T10:46:29.3762664Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27258
2022-12-01T10:46:29.3763279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27259
2022-12-01T10:46:29.3763888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3764340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3764888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3765356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3765931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3766356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3766931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3767402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3767840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3768340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3769023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3769689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3770212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3770729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3771604Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3772212Z   warnings.warn(
2022-12-01T10:46:29.3772945Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3773479Z   warnings.warn(
2022-12-01T10:46:29.3774244Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3775048Z   warnings.warn(
2022-12-01T10:46:29.3775789Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3776334Z   warnings.warn(
2022-12-01T10:46:29.3776584Z dist init r=0, world=2
2022-12-01T10:46:29.3776837Z dist init r=1, world=2
2022-12-01T10:46:29.3777058Z ok (3.911s)
2022-12-01T10:46:29.3777569Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27341
2022-12-01T10:46:29.3778169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27342
2022-12-01T10:46:29.3778761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3779208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3779797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3780263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3780827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3781473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3782056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3782507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3782962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3783456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3784119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3784792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3785315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3785788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3786650Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3787178Z   warnings.warn(
2022-12-01T10:46:29.3788020Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3788577Z   warnings.warn(
2022-12-01T10:46:29.3789344Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3789952Z   warnings.warn(
2022-12-01T10:46:29.3790710Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3791258Z   warnings.warn(
2022-12-01T10:46:29.3791506Z dist init r=0, world=2
2022-12-01T10:46:29.3791739Z dist init r=1, world=2
2022-12-01T10:46:29.3791977Z ok (3.911s)
2022-12-01T10:46:29.3792486Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27424
2022-12-01T10:46:29.3793068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27425
2022-12-01T10:46:29.3793683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3794128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3794702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3795156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3795731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3796175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3796733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3797200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3797651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3798146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3798782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3799470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3799997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3800470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3801319Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3801865Z   warnings.warn(
2022-12-01T10:46:29.3802618Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3803151Z   warnings.warn(
2022-12-01T10:46:29.3803381Z dist init r=0, world=2
2022-12-01T10:46:29.3803635Z dist init r=1, world=2
2022-12-01T10:46:29.3803871Z ok (4.011s)
2022-12-01T10:46:29.3804364Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27507
2022-12-01T10:46:29.3804964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27508
2022-12-01T10:46:29.3805637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3806128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3806707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3807182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3807761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3808189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3808814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3809284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3809743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3810226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3810886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3811571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3812075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3812545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3813401Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3813954Z   warnings.warn(
2022-12-01T10:46:29.3815010Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3815563Z   warnings.warn(
2022-12-01T10:46:29.3816337Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3816884Z   warnings.warn(
2022-12-01T10:46:29.3817620Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3818152Z   warnings.warn(
2022-12-01T10:46:29.3818402Z dist init r=1, world=2
2022-12-01T10:46:29.3818655Z dist init r=0, world=2
2022-12-01T10:46:29.3818875Z ok (4.011s)
2022-12-01T10:46:29.3819288Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3820036Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27590
2022-12-01T10:46:29.3820562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27591
2022-12-01T10:46:29.3821167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3821613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3822184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3822717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3823313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3823830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3824386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3824854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3825307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3825807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3826444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3827135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3827656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3828129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3828976Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3829521Z   warnings.warn(
2022-12-01T10:46:29.3830268Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3830800Z   warnings.warn(
2022-12-01T10:46:29.3831546Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3832101Z   warnings.warn(
2022-12-01T10:46:29.3832862Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3833405Z   warnings.warn(
2022-12-01T10:46:29.3833636Z dist init r=1, world=2
2022-12-01T10:46:29.3833888Z dist init r=0, world=2
2022-12-01T10:46:29.3834127Z ok (3.911s)
2022-12-01T10:46:29.3834523Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3835267Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27673
2022-12-01T10:46:29.3835816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27674
2022-12-01T10:46:29.3836424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3836858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3837435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3837901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3838458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3838901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3839468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3840001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3840447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3841004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3841662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3842351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3842854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3843325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3844194Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3844740Z   warnings.warn(
2022-12-01T10:46:29.3845476Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3846020Z   warnings.warn(
2022-12-01T10:46:29.3846272Z dist init r=0, world=2
2022-12-01T10:46:29.3846507Z dist init r=1, world=2
2022-12-01T10:46:29.3846744Z ok (3.911s)
2022-12-01T10:46:29.3847159Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3847888Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27756
2022-12-01T10:46:29.3848435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27757
2022-12-01T10:46:29.3849241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3849752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3850318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3850787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3851362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3851803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3852354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3852816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3853267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3853745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3854842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3855550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3856073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3856528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3857476Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3858037Z   warnings.warn(
2022-12-01T10:46:29.3858785Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3859382Z   warnings.warn(
2022-12-01T10:46:29.3860144Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3860695Z   warnings.warn(
2022-12-01T10:46:29.3861445Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.3861962Z   warnings.warn(
2022-12-01T10:46:29.3862213Z dist init r=0, world=2
2022-12-01T10:46:29.3862469Z dist init r=1, world=2
2022-12-01T10:46:29.3862690Z ok (3.912s)
2022-12-01T10:46:29.3863141Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3863889Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27839
2022-12-01T10:46:29.3864409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27840
2022-12-01T10:46:29.3864998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3865446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3866017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3866498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3867058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3867502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3868071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3868519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3868976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3869472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3870126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3870800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3871322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3871798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3872664Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3873194Z   warnings.warn(
2022-12-01T10:46:29.3873943Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3874483Z   warnings.warn(
2022-12-01T10:46:29.3875017Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3875391Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3875954Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3876412Z     p_assert(
2022-12-01T10:46:29.3876868Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3877249Z     traceback.print_stack()
2022-12-01T10:46:29.3877745Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3878106Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3878662Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3879056Z     p_assert(
2022-12-01T10:46:29.3879527Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3879889Z     traceback.print_stack()
2022-12-01T10:46:29.3880380Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3880763Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3881301Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3881695Z     p_assert(
2022-12-01T10:46:29.3882163Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3882541Z     traceback.print_stack()
2022-12-01T10:46:29.3883016Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3883389Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3883950Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3884326Z     p_assert(
2022-12-01T10:46:29.3884792Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3885173Z     traceback.print_stack()
2022-12-01T10:46:29.3885422Z dist init r=1, world=2
2022-12-01T10:46:29.3885700Z Communication hook should not be None
2022-12-01T10:46:29.3886019Z Communication hook state should not be None
2022-12-01T10:46:29.3886288Z dist init r=0, world=2
2022-12-01T10:46:29.3886566Z Communication hook should not be None
2022-12-01T10:46:29.3886883Z Communication hook state should not be None
2022-12-01T10:46:29.3887159Z ok (3.912s)
2022-12-01T10:46:29.3887587Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3888343Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27922
2022-12-01T10:46:29.3888865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27923
2022-12-01T10:46:29.3889453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3889908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3890480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3890945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3891505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3891952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3892522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3893025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3893488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3894041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3895037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3895712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3896233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3896707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3897576Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3898110Z   warnings.warn(
2022-12-01T10:46:29.3898866Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3899405Z   warnings.warn(
2022-12-01T10:46:29.3899887Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3900251Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3900811Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3901207Z     p_assert(
2022-12-01T10:46:29.3901655Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3902041Z     traceback.print_stack()
2022-12-01T10:46:29.3902539Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3902921Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3903460Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3903857Z     p_assert(
2022-12-01T10:46:29.3904322Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3904682Z     traceback.print_stack()
2022-12-01T10:46:29.3905174Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3905551Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3906089Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3906489Z     p_assert(
2022-12-01T10:46:29.3906959Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3907343Z     traceback.print_stack()
2022-12-01T10:46:29.3907822Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3908199Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3908793Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3909177Z     p_assert(
2022-12-01T10:46:29.3909645Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3910021Z     traceback.print_stack()
2022-12-01T10:46:29.3910287Z dist init r=1, world=2
2022-12-01T10:46:29.3910548Z Communication hook should not be None
2022-12-01T10:46:29.3910869Z Communication hook state should not be None
2022-12-01T10:46:29.3911261Z dist init r=0, world=2
2022-12-01T10:46:29.3911536Z Communication hook should not be None
2022-12-01T10:46:29.3911857Z Communication hook state should not be None
2022-12-01T10:46:29.3912205Z ok (3.911s)
2022-12-01T10:46:29.3912643Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3913401Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28005
2022-12-01T10:46:29.3913924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28006
2022-12-01T10:46:29.3914528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3914964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3915541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3916013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3916579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3917023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3917591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3918057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3918493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3918988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3919649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3920342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3920850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3921320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3922184Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3922735Z   warnings.warn(
2022-12-01T10:46:29.3923473Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3924015Z   warnings.warn(
2022-12-01T10:46:29.3924497Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3924865Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3925422Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3925817Z     p_assert(
2022-12-01T10:46:29.3926283Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3926644Z     traceback.print_stack()
2022-12-01T10:46:29.3927141Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3927519Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3928057Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3928446Z     p_assert(
2022-12-01T10:46:29.3928976Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3929349Z     traceback.print_stack()
2022-12-01T10:46:29.3929912Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3930295Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3930846Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3931225Z     p_assert(
2022-12-01T10:46:29.3931690Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3932070Z     traceback.print_stack()
2022-12-01T10:46:29.3932544Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3932918Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3933480Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3933877Z     p_assert(
2022-12-01T10:46:29.3934577Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3934967Z     traceback.print_stack()
2022-12-01T10:46:29.3935233Z dist init r=1, world=2
2022-12-01T10:46:29.3935497Z Communication hook should not be None
2022-12-01T10:46:29.3935818Z Communication hook state should not be None
2022-12-01T10:46:29.3936105Z dist init r=0, world=2
2022-12-01T10:46:29.3936364Z Communication hook should not be None
2022-12-01T10:46:29.3936680Z Communication hook state should not be None
2022-12-01T10:46:29.3937137Z ok (3.912s)
2022-12-01T10:46:29.3937571Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3938330Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28088
2022-12-01T10:46:29.3938850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28089
2022-12-01T10:46:29.3939460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3939891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3940462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3940931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3941510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3941937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3942505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3942973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3943411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3943906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3944563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3945254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3945760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3946234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3947183Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3947800Z   warnings.warn(
2022-12-01T10:46:29.3948529Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3949068Z   warnings.warn(
2022-12-01T10:46:29.3949548Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3949931Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3950404Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3950777Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3951335Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3951714Z     p_assert(
2022-12-01T10:46:29.3952187Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3952568Z     traceback.print_stack()
2022-12-01T10:46:29.3953107Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3953504Z     p_assert(
2022-12-01T10:46:29.3953972Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3954348Z     traceback.print_stack()
2022-12-01T10:46:29.3954823Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3955204Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3955760Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3956140Z     p_assert(
2022-12-01T10:46:29.3956603Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3956982Z     traceback.print_stack()
2022-12-01T10:46:29.3957475Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3957834Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3958390Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3958787Z     p_assert(
2022-12-01T10:46:29.3959233Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3959612Z     traceback.print_stack()
2022-12-01T10:46:29.3959879Z dist init r=1, world=2
2022-12-01T10:46:29.3960144Z Communication hook should not be None
2022-12-01T10:46:29.3960731Z Communication hook state should not be None
2022-12-01T10:46:29.3961092Z dist init r=0, world=2
2022-12-01T10:46:29.3961358Z Communication hook should not be None
2022-12-01T10:46:29.3961675Z Communication hook state should not be None
2022-12-01T10:46:29.3961952Z ok (3.912s)
2022-12-01T10:46:29.3962396Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3963129Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28171
2022-12-01T10:46:29.3963653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28172
2022-12-01T10:46:29.3964264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3964693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3965350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3965881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3966460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3966882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3967453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3967916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3968371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3968851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3969513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3970200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3970705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3971177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3972035Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3972589Z   warnings.warn(
2022-12-01T10:46:29.3973324Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3973864Z   warnings.warn(
2022-12-01T10:46:29.3974769Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3975176Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3975660Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3976034Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3976588Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3976969Z     p_assert(
2022-12-01T10:46:29.3977505Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.3977898Z     p_assert(
2022-12-01T10:46:29.3978375Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3978739Z     traceback.print_stack()
2022-12-01T10:46:29.3979233Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3979621Z     traceback.print_stack()
2022-12-01T10:46:29.3980097Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3980476Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3981036Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3981427Z     p_assert(
2022-12-01T10:46:29.3981877Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3982252Z     traceback.print_stack()
2022-12-01T10:46:29.3982746Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3983196Z     return func(*args, **kwargs)
2022-12-01T10:46:29.3983772Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.3984240Z     p_assert(
2022-12-01T10:46:29.3984689Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.3985068Z     traceback.print_stack()
2022-12-01T10:46:29.3985339Z dist init r=1, world=2
2022-12-01T10:46:29.3985622Z Communication hook should not be None
2022-12-01T10:46:29.3985924Z Communication hook state should not be None
2022-12-01T10:46:29.3986208Z dist init r=0, world=2
2022-12-01T10:46:29.3986482Z Communication hook should not be None
2022-12-01T10:46:29.3986783Z Communication hook state should not be None
2022-12-01T10:46:29.3987056Z ok (3.912s)
2022-12-01T10:46:29.3987510Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.3988247Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28254
2022-12-01T10:46:29.3988773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28255
2022-12-01T10:46:29.3989378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3989824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3990380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3990851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3991430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.3991859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.3992434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.3992903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.3993360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.3993841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.3994493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3995188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.3995708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.3996162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.3997022Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3997577Z   warnings.warn(
2022-12-01T10:46:29.3998326Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.3998850Z   warnings.warn(
2022-12-01T10:46:29.3999335Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.3999717Z     return func(*args, **kwargs)
2022-12-01T10:46:29.4000193Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.4000632Z     return func(*args, **kwargs)
2022-12-01T10:46:29.4001201Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.4001668Z     p_assert(
2022-12-01T10:46:29.4002119Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.4002500Z     traceback.print_stack()
2022-12-01T10:46:29.4003062Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3270, in _post_backward_hook
2022-12-01T10:46:29.4003437Z     p_assert(
2022-12-01T10:46:29.4003902Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.4004283Z     traceback.print_stack()
2022-12-01T10:46:29.4004759Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.4005146Z     return func(*args, **kwargs)
2022-12-01T10:46:29.4005707Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.4006101Z     p_assert(
2022-12-01T10:46:29.4006548Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.4006923Z     traceback.print_stack()
2022-12-01T10:46:29.4007414Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
2022-12-01T10:46:29.4007777Z     return func(*args, **kwargs)
2022-12-01T10:46:29.4008331Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 3274, in _post_backward_hook
2022-12-01T10:46:29.4008725Z     p_assert(
2022-12-01T10:46:29.4009235Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 147, in p_assert
2022-12-01T10:46:29.4009600Z     traceback.print_stack()
2022-12-01T10:46:29.4009868Z dist init r=0, world=2
2022-12-01T10:46:29.4010150Z Communication hook should not be None
2022-12-01T10:46:29.4010451Z Communication hook state should not be None
2022-12-01T10:46:29.4010741Z dist init r=1, world=2
2022-12-01T10:46:29.4011017Z Communication hook should not be None
2022-12-01T10:46:29.4011318Z Communication hook state should not be None
2022-12-01T10:46:29.4011593Z ok (4.012s)
2022-12-01T10:46:29.4012103Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28337
2022-12-01T10:46:29.4012710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28338
2022-12-01T10:46:29.4013307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4013755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4014543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4015015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4015607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4016052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4016622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4017072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4017528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4018023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4018742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4019450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4020043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4020517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4021367Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4021921Z   warnings.warn(
2022-12-01T10:46:29.4022674Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4023213Z   warnings.warn(
2022-12-01T10:46:29.4023963Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4024509Z   warnings.warn(
2022-12-01T10:46:29.4025266Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4025804Z   warnings.warn(
2022-12-01T10:46:29.4026033Z dist init r=1, world=2
2022-12-01T10:46:29.4026286Z dist init r=0, world=2
2022-12-01T10:46:29.4026530Z ok (4.011s)
2022-12-01T10:46:29.4027027Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28420
2022-12-01T10:46:29.4027629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28421
2022-12-01T10:46:29.4028245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4028700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4029257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4029729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4030308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4030744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4031328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4031782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4032239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4032790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4033449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4034118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4034636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4035105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4036027Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4036613Z   warnings.warn(
2022-12-01T10:46:29.4037363Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4037900Z   warnings.warn(
2022-12-01T10:46:29.4038128Z dist init r=1, world=2
2022-12-01T10:46:29.4038377Z dist init r=0, world=2
2022-12-01T10:46:29.4038611Z ok (3.911s)
2022-12-01T10:46:29.4039124Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28503
2022-12-01T10:46:29.4039718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28504
2022-12-01T10:46:29.4040325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4040777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4041330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4041798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4042375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4042819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4043370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4043831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4044288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4044784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4045429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4046116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4046638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4047091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4047964Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4048511Z   warnings.warn(
2022-12-01T10:46:29.4049258Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4049793Z   warnings.warn(
2022-12-01T10:46:29.4050536Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4051085Z   warnings.warn(
2022-12-01T10:46:29.4051845Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4052376Z   warnings.warn(
2022-12-01T10:46:29.4052658Z dist init r=0, world=2
2022-12-01T10:46:29.4052919Z dist init r=1, world=2
2022-12-01T10:46:29.4053155Z ok (3.911s)
2022-12-01T10:46:29.4053650Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28586
2022-12-01T10:46:29.4054304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28587
2022-12-01T10:46:29.4055095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4055526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4056096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4056562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4057140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4057566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4058143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4058602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4059036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4059528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4060180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4060864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4061369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4061840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4062701Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4063245Z   warnings.warn(
2022-12-01T10:46:29.4063974Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4064510Z   warnings.warn(
2022-12-01T10:46:29.4065274Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4065821Z   warnings.warn(
2022-12-01T10:46:29.4066555Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4067095Z   warnings.warn(
2022-12-01T10:46:29.4067338Z dist init r=0, world=2
2022-12-01T10:46:29.4067586Z dist init r=1, world=2
2022-12-01T10:46:29.4067807Z ok (3.911s)
2022-12-01T10:46:29.4068312Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28669
2022-12-01T10:46:29.4068908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28670
2022-12-01T10:46:29.4069579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4070043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4070688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4071155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4071715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4072156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4072725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4073172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4073622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4074117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4074770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4075439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4075958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4076429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4077293Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4077822Z   warnings.warn(
2022-12-01T10:46:29.4078582Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4079118Z   warnings.warn(
2022-12-01T10:46:29.4079368Z dist init r=0, world=2
2022-12-01T10:46:29.4079598Z dist init r=1, world=2
2022-12-01T10:46:29.4079836Z ok (3.911s)
2022-12-01T10:46:29.4080346Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28752
2022-12-01T10:46:29.4080929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28753
2022-12-01T10:46:29.4081534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4082010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4082584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4083034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4083614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4084060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4084612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4085079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4085529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4086021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4086726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4087428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4088010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4088480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4089331Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4089873Z   warnings.warn(
2022-12-01T10:46:29.4090621Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:46:29.4091159Z   warnings.warn(
2022-12-01T10:46:29.4091896Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4092440Z   warnings.warn(
2022-12-01T10:46:29.4093199Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:46:29.4093733Z   warnings.warn(
2022-12-01T10:46:29.4093959Z dist init r=0, world=2
2022-12-01T10:46:29.4094207Z dist init r=1, world=2
2022-12-01T10:46:29.4094598Z ok (4.011s)
2022-12-01T10:46:29.4094979Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.4095696Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28835
2022-12-01T10:46:29.4096230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28836
2022-12-01T10:46:29.4096818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4097265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4097840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4098304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4098861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4099305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4099875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4100339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4100780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4101272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4101928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4102597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4103117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4103584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4104017Z dist init r=1, world=2
2022-12-01T10:46:29.4104267Z dist init r=0, world=2
2022-12-01T10:46:29.4104504Z ok (3.410s)
2022-12-01T10:46:29.4104966Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.4105660Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28914
2022-12-01T10:46:29.4106188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28915
2022-12-01T10:46:29.4106793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4107244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4107797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4108268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4108885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4109317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4109890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4110352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4110802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4111279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4111934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4112836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4113469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4113926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4114274Z dist init r=0, world=2
2022-12-01T10:46:29.4114524Z dist init r=1, world=2
2022-12-01T10:46:29.4114745Z ok (3.410s)
2022-12-01T10:46:29.4115146Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.4115869Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28993
2022-12-01T10:46:29.4116399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28994
2022-12-01T10:46:29.4116985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4117437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4118009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4118465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4119038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4119476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4120044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4120491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4120939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4121500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4122150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4122897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4123417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4123887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4124224Z dist init r=1, world=2
2022-12-01T10:46:29.4124473Z dist init r=0, world=2
2022-12-01T10:46:29.4124711Z ok (3.510s)
2022-12-01T10:46:29.4125096Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.4125817Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29072
2022-12-01T10:46:29.4126346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29073
2022-12-01T10:46:29.4126954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4127385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4127954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4128417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4128991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4129413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4129984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4130447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4130880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4131375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4132025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4132708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4133208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4133674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4134027Z dist init r=1, world=2
2022-12-01T10:46:29.4134260Z dist init r=0, world=2
2022-12-01T10:46:29.4134725Z ok (3.410s)
2022-12-01T10:46:29.4135131Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.4135852Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29151
2022-12-01T10:46:29.4136365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29152
2022-12-01T10:46:29.4136966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4137415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4137971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4138440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4139097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4139556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4140180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4140642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4141093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4141584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4142219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4142903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4143430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4143886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4144243Z dist init r=0, world=2
2022-12-01T10:46:29.4144493Z dist init r=1, world=2
2022-12-01T10:46:29.4144729Z ok (3.510s)
2022-12-01T10:46:29.4145119Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2022-12-01T10:46:29.4145842Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29230
2022-12-01T10:46:29.4146372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29231
2022-12-01T10:46:29.4146958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4147409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4147985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4148454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4149020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:46:29.4149463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:46:29.4150034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:46:29.4150479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:46:29.4150927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:46:29.4151420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:46:29.4152078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4152748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:46:29.4153273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:46:29.4153740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:46:29.4154095Z dist init r=0, world=2
2022-12-01T10:46:29.4154327Z dist init r=1, world=2
2022-12-01T10:46:29.4154564Z ok (3.410s)
2022-12-01T10:46:29.4154710Z 
2022-12-01T10:46:29.4154983Z ----------------------------------------------------------------------
2022-12-01T10:46:29.4155297Z Ran 27 tests in 104.882s
2022-12-01T10:46:29.4155458Z 
2022-12-01T10:46:29.4155551Z OK
2022-12-01T10:46:29.4155683Z 
2022-12-01T10:46:29.4155806Z Generating XML reports...
2022-12-01T10:46:29.4156466Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221201104444.xml
2022-12-01T10:46:29.4156848Z 
2022-12-01T10:46:29.4157360Z ##[endgroup]
2022-12-01T10:46:29.4157974Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_n24fq9c8)
2022-12-01T10:46:29.4158336Z 
2022-12-01T10:46:29.4158588Z Running distributed/fsdp/test_wrap ... [2022-12-01 10:46:29.373705]
2022-12-01T10:46:29.4159224Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:46:29.374034]
2022-12-01T10:47:59.3922819Z 
2022-12-01T10:47:59.3923307Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2022-12-01T10:47:59.3924251Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_hl42573g)
2022-12-01T10:47:59.3926612Z 
2022-12-01T10:47:59.3927028Z Running tests...
2022-12-01T10:47:59.3927832Z ----------------------------------------------------------------------
2022-12-01T10:47:59.3928383Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-12-01T10:47:59.3928755Z   test_always_wrap (__main__.TestAutoWrap)
2022-12-01T10:47:59.3929139Z Test to ensure that if `always_wrap_policy` is ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:47:59.3929524Z ok (1.569s)
2022-12-01T10:47:59.3931813Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.3932735Z   warnings.warn(
2022-12-01T10:47:59.3932910Z ok (0.005s)
2022-12-01T10:47:59.3933288Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.004s)
2022-12-01T10:47:59.3933677Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-12-01T10:47:59.3935261Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3937412Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3939861Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3940987Z ok (0.003s)
2022-12-01T10:47:59.3941610Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-12-01T10:47:59.3942778Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... ok (0.002s)
2022-12-01T10:47:59.3943184Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-12-01T10:47:59.3943664Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... ok (0.002s)
2022-12-01T10:47:59.3944476Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-12-01T10:47:59.3945619Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... ok (0.003s)
2022-12-01T10:47:59.3946626Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-12-01T10:47:59.3948821Z Test to ensure force-leaf modules are not wrapped. ... [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3951150Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3953292Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3954488Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3955090Z ok (0.002s)
2022-12-01T10:47:59.3955674Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.3956514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:47:59.3957491Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.3957971Z   warnings.warn(
2022-12-01T10:47:59.3958216Z ok (0.450s)
2022-12-01T10:47:59.3958768Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.3959608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:47:59.3960653Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3961884Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3962482Z ok (0.050s)
2022-12-01T10:47:59.3962915Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... ok (0.002s)
2022-12-01T10:47:59.3963570Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... ok (0.002s)
2022-12-01T10:47:59.3964400Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.3965373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:47:59.3965877Z ok (0.045s)
2022-12-01T10:47:59.3966431Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.3967282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:47:59.3967692Z ok (0.042s)
2022-12-01T10:47:59.3968387Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.3969188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:47:59.3970287Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3971622Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3972939Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3978385Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3979650Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3980885Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3982113Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3983390Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3984681Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3986106Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3987432Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3988763Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3990091Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3991399Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3992727Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3994051Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3994698Z ok (0.072s)
2022-12-01T10:47:59.3995308Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.3996200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:47:59.3996694Z ok (0.045s)
2022-12-01T10:47:59.3997002Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.003s)
2022-12-01T10:47:59.3998269Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.3999776Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4001114Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4002492Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4003823Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4004467Z ok (0.018s)
2022-12-01T10:47:59.4004784Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2022-12-01T10:47:59.4005874Z Tests the ``transformer_auto_wrap_policy``. ... [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4006563Z ok (0.020s)
2022-12-01T10:47:59.4006903Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... ok (0.002s)
2022-12-01T10:47:59.4008041Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4009444Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4010197Z ok (0.002s)
2022-12-01T10:47:59.4010430Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.002s)
2022-12-01T10:47:59.4010900Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.002s)
2022-12-01T10:47:59.4011334Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-12-01T10:47:59.4011865Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29376
2022-12-01T10:47:59.4012441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29377
2022-12-01T10:47:59.4013091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4013569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4014157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4015044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4015623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4016095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4016681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4017269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4017756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4018334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4019032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4019798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4020352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4020833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4022175Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4023153Z   warnings.warn(
2022-12-01T10:47:59.4024342Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4025145Z   warnings.warn(
2022-12-01T10:47:59.4025414Z dist init r=0, world=2
2022-12-01T10:47:59.4025878Z dist init r=1, world=2
2022-12-01T10:47:59.4026406Z ok (3.512s)
2022-12-01T10:47:59.4026815Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-12-01T10:47:59.4027496Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29455
2022-12-01T10:47:59.4028119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29456
2022-12-01T10:47:59.4028769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4029343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4046276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4046780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4047351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4047814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4048396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4048852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4049311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4049810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4050473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4051149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4051672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4052263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4053577Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4055604Z   warnings.warn(
2022-12-01T10:47:59.4056796Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4057608Z   warnings.warn(
2022-12-01T10:47:59.4057867Z dist init r=0, world=2
2022-12-01T10:47:59.4058113Z dist init r=1, world=2
2022-12-01T10:47:59.4058361Z ok (3.410s)
2022-12-01T10:47:59.4058760Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-12-01T10:47:59.4059368Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29534
2022-12-01T10:47:59.4059920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29535
2022-12-01T10:47:59.4060578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4061055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4061650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4062152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4062766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4063237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4063823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4064323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4064795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4065320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4065996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4066727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4067289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4067775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4068147Z dist init r=1, world=2
2022-12-01T10:47:59.4068406Z dist init r=0, world=2
2022-12-01T10:47:59.4068652Z ok (3.510s)
2022-12-01T10:47:59.4069029Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-12-01T10:47:59.4069635Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29613
2022-12-01T10:47:59.4070204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29614
2022-12-01T10:47:59.4070946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4071441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4072148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4072643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4073238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4073710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4074316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4074810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4075277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4075805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4076502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4077216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4077767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4078271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4079564Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4080376Z   warnings.warn(
2022-12-01T10:47:59.4081532Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4082340Z   warnings.warn(
2022-12-01T10:47:59.4082597Z dist init r=0, world=2
2022-12-01T10:47:59.4082856Z dist init r=1, world=2
2022-12-01T10:47:59.4083088Z ok (3.410s)
2022-12-01T10:47:59.4083488Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-12-01T10:47:59.4084097Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29692
2022-12-01T10:47:59.4084655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29693
2022-12-01T10:47:59.4085297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4085767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4086372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4086849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4087460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4087932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4088580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4089093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4089624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4090144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4090822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4091546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4092097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4092597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4092956Z dist init r=0, world=2
2022-12-01T10:47:59.4093218Z dist init r=1, world=2
2022-12-01T10:47:59.4093465Z ok (3.410s)
2022-12-01T10:47:59.4094117Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29771
2022-12-01T10:47:59.4095150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29772
2022-12-01T10:47:59.4095744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4096192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4096746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4097219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4097794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4098221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4098785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4099245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4099695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4100177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4100831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4101529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4102103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4102563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4103744Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4104496Z   warnings.warn(
2022-12-01T10:47:59.4105696Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4106518Z   warnings.warn(
2022-12-01T10:47:59.4107267Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4107816Z   warnings.warn(
2022-12-01T10:47:59.4108571Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4109114Z   warnings.warn(
2022-12-01T10:47:59.4109865Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4110415Z   warnings.warn(
2022-12-01T10:47:59.4111180Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4111730Z   warnings.warn(
2022-12-01T10:47:59.4111958Z dist init r=1, world=2
2022-12-01T10:47:59.4112208Z dist init r=0, world=2
2022-12-01T10:47:59.4112447Z ok (3.917s)
2022-12-01T10:47:59.4113073Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29854
2022-12-01T10:47:59.4113807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29855
2022-12-01T10:47:59.4114426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4114913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4115476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4115956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4116541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4117000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4117591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4118047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4118505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4119013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4119654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4120399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4120923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4121399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4122317Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4122878Z   warnings.warn(
2022-12-01T10:47:59.4123638Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4124232Z   warnings.warn(
2022-12-01T10:47:59.4124984Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4125534Z   warnings.warn(
2022-12-01T10:47:59.4126306Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4126869Z   warnings.warn(
2022-12-01T10:47:59.4127099Z dist init r=0, world=2
2022-12-01T10:47:59.4127350Z dist init r=1, world=2
2022-12-01T10:47:59.4127593Z ok (4.012s)
2022-12-01T10:47:59.4128203Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29937
2022-12-01T10:47:59.4128931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29938
2022-12-01T10:47:59.4129548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4130000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4130580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4131035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4131607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4132053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4132602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4133066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4133517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4134018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4134946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4135611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4136140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4136619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4137844Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4140259Z   warnings.warn(
2022-12-01T10:47:59.4141513Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4142342Z   warnings.warn(
2022-12-01T10:47:59.4143109Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4143679Z   warnings.warn(
2022-12-01T10:47:59.4144417Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4144981Z   warnings.warn(
2022-12-01T10:47:59.4145779Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4146347Z   warnings.warn(
2022-12-01T10:47:59.4147107Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4147661Z   warnings.warn(
2022-12-01T10:47:59.4147914Z dist init r=0, world=2
2022-12-01T10:47:59.4148168Z dist init r=1, world=2
2022-12-01T10:47:59.4148404Z ok (4.012s)
2022-12-01T10:47:59.4149041Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30020
2022-12-01T10:47:59.4149766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30021
2022-12-01T10:47:59.4150384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4150852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4151457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4152062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4152603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4153032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4153594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4154061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4154517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4154998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4155660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4156350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4156852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4157320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4158293Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4158846Z   warnings.warn(
2022-12-01T10:47:59.4159664Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4160199Z   warnings.warn(
2022-12-01T10:47:59.4160960Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4161503Z   warnings.warn(
2022-12-01T10:47:59.4162247Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4162779Z   warnings.warn(
2022-12-01T10:47:59.4163022Z dist init r=1, world=2
2022-12-01T10:47:59.4163273Z dist init r=0, world=2
2022-12-01T10:47:59.4163491Z ok (4.013s)
2022-12-01T10:47:59.4164122Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30103
2022-12-01T10:47:59.4164828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30104
2022-12-01T10:47:59.4165442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4165872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4166452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4166922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4167484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4167933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4168502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4168969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4169401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4169899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4170558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4171231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4171754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4172230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4173447Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4174196Z   warnings.warn(
2022-12-01T10:47:59.4175731Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4176548Z   warnings.warn(
2022-12-01T10:47:59.4177310Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4177835Z   warnings.warn(
2022-12-01T10:47:59.4178569Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4179098Z   warnings.warn(
2022-12-01T10:47:59.4179864Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4180411Z   warnings.warn(
2022-12-01T10:47:59.4181155Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4181687Z   warnings.warn(
2022-12-01T10:47:59.4181928Z dist init r=0, world=2
2022-12-01T10:47:59.4182174Z dist init r=1, world=2
2022-12-01T10:47:59.4182393Z ok (4.012s)
2022-12-01T10:47:59.4183026Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30186
2022-12-01T10:47:59.4183732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30187
2022-12-01T10:47:59.4184348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4184778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4185347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4185814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4186372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4186809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4187379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4187842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4188273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4188773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4189426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4190166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4190622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4191091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4192014Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4192604Z   warnings.warn(
2022-12-01T10:47:59.4193352Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4193932Z   warnings.warn(
2022-12-01T10:47:59.4194648Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4195186Z   warnings.warn(
2022-12-01T10:47:59.4195925Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4196461Z   warnings.warn(
2022-12-01T10:47:59.4196711Z dist init r=0, world=2
2022-12-01T10:47:59.4196941Z dist init r=1, world=2
2022-12-01T10:47:59.4197173Z ok (3.913s)
2022-12-01T10:47:59.4197796Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30269
2022-12-01T10:47:59.4198506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30270
2022-12-01T10:47:59.4199097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4199546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4200122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4200575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4201154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4201594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4202163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4202609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4203061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4203559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4204215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4204933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4205416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4205883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4207098Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4207829Z   warnings.warn(
2022-12-01T10:47:59.4208983Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4209785Z   warnings.warn(
2022-12-01T10:47:59.4210537Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4211076Z   warnings.warn(
2022-12-01T10:47:59.4211802Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4212336Z   warnings.warn(
2022-12-01T10:47:59.4213096Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4213638Z   warnings.warn(
2022-12-01T10:47:59.4214634Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4215210Z   warnings.warn(
2022-12-01T10:47:59.4215413Z dist init r=0, world=2
2022-12-01T10:47:59.4215643Z dist init r=1, world=2
2022-12-01T10:47:59.4215878Z ok (4.012s)
2022-12-01T10:47:59.4216507Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30352
2022-12-01T10:47:59.4217209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30353
2022-12-01T10:47:59.4217817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4218264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4218837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4219290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4219914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4220358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4220928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4221372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4221828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4222325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4222980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4223650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4224172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4224642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4225616Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4226214Z   warnings.warn(
2022-12-01T10:47:59.4226964Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4227499Z   warnings.warn(
2022-12-01T10:47:59.4228263Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4228784Z   warnings.warn(
2022-12-01T10:47:59.4229573Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4230086Z   warnings.warn(
2022-12-01T10:47:59.4230288Z dist init r=1, world=2
2022-12-01T10:47:59.4230534Z dist init r=0, world=2
2022-12-01T10:47:59.4231410Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4232628Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4233849Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:47:59.4234426Z ok (3.913s)
2022-12-01T10:47:59.4235035Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30435
2022-12-01T10:47:59.4235739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30436
2022-12-01T10:47:59.4236349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4236782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4237360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4237828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4238408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4238835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4239399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4239859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4240296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4240796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4241509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4242210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4242778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4243245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4243593Z dist init r=1, world=2
2022-12-01T10:47:59.4243843Z dist init r=0, world=2
2022-12-01T10:47:59.4244066Z ok (3.411s)
2022-12-01T10:47:59.4244692Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30514
2022-12-01T10:47:59.4245407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30515
2022-12-01T10:47:59.4246005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4246445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4247016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4247482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4248040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4248485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4249051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4249516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4249952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4250444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4251105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4251773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4252294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4252757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4253618Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4254146Z   warnings.warn(
2022-12-01T10:47:59.4255107Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4255646Z   warnings.warn(
2022-12-01T10:47:59.4256416Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4256944Z   warnings.warn(
2022-12-01T10:47:59.4257700Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4258340Z   warnings.warn(
2022-12-01T10:47:59.4258596Z dist init r=0, world=2
2022-12-01T10:47:59.4258827Z dist init r=1, world=2
2022-12-01T10:47:59.4259063Z ok (4.012s)
2022-12-01T10:47:59.4259765Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30597
2022-12-01T10:47:59.4260454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30598
2022-12-01T10:47:59.4261073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4261523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4262096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4262551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4263131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4263576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4264144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4264590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4265039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4265538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4266177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4266872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4267400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4267872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4268209Z dist init r=0, world=2
2022-12-01T10:47:59.4268457Z dist init r=1, world=2
2022-12-01T10:47:59.4268695Z ok (3.411s)
2022-12-01T10:47:59.4269304Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30676
2022-12-01T10:47:59.4270006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30677
2022-12-01T10:47:59.4270617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4271065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4271621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4272095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4272668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4273109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4273657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4274116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4274564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4275104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4275776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4276538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4277056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4277511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4278372Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4278913Z   warnings.warn(
2022-12-01T10:47:59.4279669Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4280191Z   warnings.warn(
2022-12-01T10:47:59.4280951Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4281493Z   warnings.warn(
2022-12-01T10:47:59.4282244Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4282760Z   warnings.warn(
2022-12-01T10:47:59.4283001Z dist init r=1, world=2
2022-12-01T10:47:59.4283241Z dist init r=0, world=2
2022-12-01T10:47:59.4283454Z ok (4.013s)
2022-12-01T10:47:59.4284069Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30759
2022-12-01T10:47:59.4284779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30760
2022-12-01T10:47:59.4285387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4285820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4286392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4286857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4287433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4287859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4288424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4288887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4289320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4289820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4290522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4291165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4291727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4292212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4292568Z dist init r=0, world=2
2022-12-01T10:47:59.4292876Z dist init r=1, world=2
2022-12-01T10:47:59.4293096Z ok (3.411s)
2022-12-01T10:47:59.4293720Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30838
2022-12-01T10:47:59.4294686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30839
2022-12-01T10:47:59.4295262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4295710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4296287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4296752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4297318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4297760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4298326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4298782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4299226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4299710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4300363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4301037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4301561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4302029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4302895Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4303420Z   warnings.warn(
2022-12-01T10:47:59.4304172Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4304712Z   warnings.warn(
2022-12-01T10:47:59.4305475Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4306007Z   warnings.warn(
2022-12-01T10:47:59.4306765Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4307303Z   warnings.warn(
2022-12-01T10:47:59.4307544Z dist init r=1, world=2
2022-12-01T10:47:59.4307778Z dist init r=0, world=2
2022-12-01T10:47:59.4308016Z ok (4.013s)
2022-12-01T10:47:59.4308729Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30921
2022-12-01T10:47:59.4309427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30922
2022-12-01T10:47:59.4310127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4310575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4311148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4311596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4312170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4312610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4313177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4313621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4314077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4314575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4315254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4315902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4316425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4316892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4317228Z dist init r=1, world=2
2022-12-01T10:47:59.4317481Z dist init r=0, world=2
2022-12-01T10:47:59.4317718Z ok (3.511s)
2022-12-01T10:47:59.4318325Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31000
2022-12-01T10:47:59.4319035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31001
2022-12-01T10:47:59.4319645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4320139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4320697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4321161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4321747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4322230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4322787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4323226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4323678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4324207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4324823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4325508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4326091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4326555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4327471Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4328017Z   warnings.warn(
2022-12-01T10:47:59.4328765Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:47:59.4329282Z   warnings.warn(
2022-12-01T10:47:59.4330049Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4330596Z   warnings.warn(
2022-12-01T10:47:59.4331354Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:47:59.4331871Z   warnings.warn(
2022-12-01T10:47:59.4332115Z dist init r=1, world=2
2022-12-01T10:47:59.4332375Z dist init r=0, world=2
2022-12-01T10:47:59.4332578Z ok (4.013s)
2022-12-01T10:47:59.4333028Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31083
2022-12-01T10:47:59.4333575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31084
2022-12-01T10:47:59.4334184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4334891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4335443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4335908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4336484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4336908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4337473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4337935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4338370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4338869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4339524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4340212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4340711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4341178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4342488Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4343247Z   warnings.warn(
2022-12-01T10:47:59.4344427Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4345154Z   warnings.warn(
2022-12-01T10:47:59.4345402Z dist init r=1, world=2
2022-12-01T10:47:59.4345652Z dist init r=0, world=2
2022-12-01T10:47:59.4345868Z ok (3.510s)
2022-12-01T10:47:59.4346322Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31162
2022-12-01T10:47:59.4346874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31163
2022-12-01T10:47:59.4347485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4347925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4348501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4348967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4349524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:47:59.4349966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:47:59.4350529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:47:59.4350996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:47:59.4351429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:47:59.4351931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:47:59.4352585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4353273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:47:59.4353779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:47:59.4354250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:47:59.4355470Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4356222Z   warnings.warn(
2022-12-01T10:47:59.4357301Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:47:59.4358043Z   warnings.warn(
2022-12-01T10:47:59.4358290Z dist init r=0, world=2
2022-12-01T10:47:59.4358538Z dist init r=1, world=2
2022-12-01T10:47:59.4358756Z ok (3.410s)
2022-12-01T10:47:59.4358903Z 
2022-12-01T10:47:59.4359243Z ----------------------------------------------------------------------
2022-12-01T10:47:59.4359586Z Ran 46 tests in 88.125s
2022-12-01T10:47:59.4359752Z 
2022-12-01T10:47:59.4359885Z OK
2022-12-01T10:47:59.4360019Z 
2022-12-01T10:47:59.4360143Z Generating XML reports...
2022-12-01T10:47:59.4360710Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221201104630.xml
2022-12-01T10:47:59.4361410Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221201104630.xml
2022-12-01T10:47:59.4361732Z 
2022-12-01T10:47:59.4362074Z ##[endgroup]
2022-12-01T10:47:59.4362645Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_hl42573g)
2022-12-01T10:47:59.4362980Z 
2022-12-01T10:47:59.4363224Z Running distributed/test_store ... [2022-12-01 10:47:59.392857]
2022-12-01T10:47:59.4363885Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:47:59.393139]
2022-12-01T10:49:42.1124914Z 
2022-12-01T10:49:42.1125440Z Expand the folded group to see the log file of distributed/test_store
2022-12-01T10:49:42.1128399Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_zse1uhl2)
2022-12-01T10:49:42.1129018Z <unittest.suite.TestSuite tests=[<__main__.FileStoreTest testMethod=test_compare_set>, <__main__.FileStoreTest testMethod=test_set_get>]>
2022-12-01T10:49:42.1129492Z test_compare_set (__main__.FileStoreTest)
2022-12-01T10:49:42.1130012Z test_set_get (__main__.FileStoreTest)
2022-12-01T10:49:42.1130541Z <unittest.suite.TestSuite tests=[<__main__.HashStoreTest testMethod=test_compare_set>, <__main__.HashStoreTest testMethod=test_set_get>]>
2022-12-01T10:49:42.1133807Z test_compare_set (__main__.HashStoreTest)
2022-12-01T10:49:42.1134159Z test_set_get (__main__.HashStoreTest)
2022-12-01T10:49:42.1135328Z <unittest.suite.TestSuite tests=[<__main__.PrefixFileStoreTest testMethod=test_compare_set>, <__main__.PrefixFileStoreTest testMethod=test_set_get>]>
2022-12-01T10:49:42.1135872Z test_compare_set (__main__.PrefixFileStoreTest)
2022-12-01T10:49:42.1136068Z test_set_get (__main__.PrefixFileStoreTest)
2022-12-01T10:49:42.1136654Z <unittest.suite.TestSuite tests=[<__main__.PrefixStoreTest testMethod=test_get_underlying_store>]>
2022-12-01T10:49:42.1137113Z test_get_underlying_store (__main__.PrefixStoreTest)
2022-12-01T10:49:42.1137627Z <unittest.suite.TestSuite tests=[<__main__.PrefixTCPStoreTest testMethod=test_compare_set>, <__main__.PrefixTCPStoreTest testMethod=test_set_get>]>
2022-12-01T10:49:42.1138062Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-12-01T10:49:42.1138455Z test_set_get (__main__.PrefixTCPStoreTest)
2022-12-01T10:49:42.1138870Z <unittest.suite.TestSuite tests=[<__main__.PythonStoreTest testMethod=test_set_get>]>
2022-12-01T10:49:42.1139276Z test_set_get (__main__.PythonStoreTest)
2022-12-01T10:49:42.1139644Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_nominal>]>
2022-12-01T10:49:42.1139910Z test_nominal (__main__.RendezvousEnvTest)
2022-12-01T10:49:42.1140463Z <unittest.suite.TestSuite tests=[<__main__.RendezvousFileTest testMethod=test_common_errors>, <__main__.RendezvousFileTest testMethod=test_nominal>]>
2022-12-01T10:49:42.1140934Z test_common_errors (__main__.RendezvousFileTest)
2022-12-01T10:49:42.1141263Z test_nominal (__main__.RendezvousFileTest)
2022-12-01T10:49:42.1141988Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTCPTest testMethod=test_common_errors>, <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]>
2022-12-01T10:49:42.1142687Z test_common_errors (__main__.RendezvousTCPTest)
2022-12-01T10:49:42.1142977Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-12-01T10:49:42.1143659Z test_nominal (__main__.RendezvousTCPTest)
2022-12-01T10:49:42.1144136Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-12-01T10:49:42.1144724Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTest testMethod=test_unknown_handler>, <__main__.RendezvousTest testMethod=test_url_with_node_params>]>
2022-12-01T10:49:42.1145287Z test_unknown_handler (__main__.RendezvousTest)
2022-12-01T10:49:42.1145765Z test_url_with_node_params (__main__.RendezvousTest)
2022-12-01T10:49:42.1146807Z <unittest.suite.TestSuite tests=[<__main__.TCPStoreTest testMethod=test_address_already_in_use>, <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]>
2022-12-01T10:49:42.1147803Z test_address_already_in_use (__main__.TCPStoreTest)
2022-12-01T10:49:42.1148182Z test_compare_set (__main__.TCPStoreTest)
2022-12-01T10:49:42.1148543Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-12-01T10:49:42.1148967Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-12-01T10:49:42.1149370Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-12-01T10:49:42.1149868Z test_multitenancy (__main__.TCPStoreTest)
2022-12-01T10:49:42.1150206Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-12-01T10:49:42.1150504Z test_set_get (__main__.TCPStoreTest)
2022-12-01T10:49:42.1150995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1151455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1152041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1152500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1152736Z 
2022-12-01T10:49:42.1152852Z Running tests...
2022-12-01T10:49:42.1153263Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1153787Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1154237Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1154576Z ok (1.562s)
2022-12-01T10:49:42.1154721Z 
2022-12-01T10:49:42.1154992Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1155299Z Ran 1 test in 1.563s
2022-12-01T10:49:42.1155488Z 
2022-12-01T10:49:42.1155556Z OK
2022-12-01T10:49:42.1155688Z 
2022-12-01T10:49:42.1155813Z Generating XML reports...
2022-12-01T10:49:42.1156352Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201104802.xml
2022-12-01T10:49:42.1157037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1157492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1158066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1158569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1158749Z 
2022-12-01T10:49:42.1158858Z Running tests...
2022-12-01T10:49:42.1159259Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1159779Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1160217Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1160547Z ok (1.546s)
2022-12-01T10:49:42.1160696Z 
2022-12-01T10:49:42.1161041Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1161370Z Ran 1 test in 1.546s
2022-12-01T10:49:42.1161522Z 
2022-12-01T10:49:42.1161617Z OK
2022-12-01T10:49:42.1161808Z 
2022-12-01T10:49:42.1161937Z Generating XML reports...
2022-12-01T10:49:42.1162473Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201104805.xml
2022-12-01T10:49:42.1163148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1163601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1164309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1164625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1164855Z 
2022-12-01T10:49:42.1164967Z Running tests...
2022-12-01T10:49:42.1165375Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1165878Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1166343Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1166684Z ok (1.572s)
2022-12-01T10:49:42.1166829Z 
2022-12-01T10:49:42.1167090Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1167418Z Ran 1 test in 1.572s
2022-12-01T10:49:42.1167555Z 
2022-12-01T10:49:42.1167649Z OK
2022-12-01T10:49:42.1167781Z 
2022-12-01T10:49:42.1167905Z Generating XML reports...
2022-12-01T10:49:42.1168437Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201104809.xml
2022-12-01T10:49:42.1169192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1169610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1170137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1170594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1170826Z 
2022-12-01T10:49:42.1170938Z Running tests...
2022-12-01T10:49:42.1171339Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1171836Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1172287Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1172615Z ok (1.536s)
2022-12-01T10:49:42.1172760Z 
2022-12-01T10:49:42.1173023Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1173327Z Ran 1 test in 1.537s
2022-12-01T10:49:42.1173485Z 
2022-12-01T10:49:42.1173579Z OK
2022-12-01T10:49:42.1173710Z 
2022-12-01T10:49:42.1173834Z Generating XML reports...
2022-12-01T10:49:42.1174936Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201104812.xml
2022-12-01T10:49:42.1175590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1176058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1176624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1177077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1177307Z 
2022-12-01T10:49:42.1177427Z Running tests...
2022-12-01T10:49:42.1177818Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1178318Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1178796Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1179148Z ok (1.541s)
2022-12-01T10:49:42.1179400Z 
2022-12-01T10:49:42.1179689Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1179999Z Ran 1 test in 1.542s
2022-12-01T10:49:42.1180245Z 
2022-12-01T10:49:42.1180340Z OK
2022-12-01T10:49:42.1180474Z 
2022-12-01T10:49:42.1180598Z Generating XML reports...
2022-12-01T10:49:42.1181156Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201104815.xml
2022-12-01T10:49:42.1181847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1182296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1182871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1183324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1183574Z 
2022-12-01T10:49:42.1183690Z Running tests...
2022-12-01T10:49:42.1184075Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1184574Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1185050Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1185407Z ok (1.546s)
2022-12-01T10:49:42.1185554Z 
2022-12-01T10:49:42.1185820Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1186129Z Ran 1 test in 1.547s
2022-12-01T10:49:42.1186290Z 
2022-12-01T10:49:42.1186382Z OK
2022-12-01T10:49:42.1186516Z 
2022-12-01T10:49:42.1186640Z Generating XML reports...
2022-12-01T10:49:42.1187193Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201104819.xml
2022-12-01T10:49:42.1187883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1188336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1188911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1189367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1189597Z 
2022-12-01T10:49:42.1189706Z Running tests...
2022-12-01T10:49:42.1190107Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1190608Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1191036Z   test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.003s)
2022-12-01T10:49:42.1191417Z 
2022-12-01T10:49:42.1191686Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1192004Z Ran 1 test in 0.003s
2022-12-01T10:49:42.1192146Z 
2022-12-01T10:49:42.1192241Z OK
2022-12-01T10:49:42.1192374Z 
2022-12-01T10:49:42.1192501Z Generating XML reports...
2022-12-01T10:49:42.1193059Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221201104822.xml
2022-12-01T10:49:42.1193722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1194170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1194741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1195208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1195436Z 
2022-12-01T10:49:42.1195528Z Running tests...
2022-12-01T10:49:42.1195930Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1196447Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1196985Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1197340Z ok (1.556s)
2022-12-01T10:49:42.1197486Z 
2022-12-01T10:49:42.1197751Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1198134Z Ran 1 test in 1.556s
2022-12-01T10:49:42.1198277Z 
2022-12-01T10:49:42.1198369Z OK
2022-12-01T10:49:42.1198501Z 
2022-12-01T10:49:42.1198625Z Generating XML reports...
2022-12-01T10:49:42.1199197Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201104824.xml
2022-12-01T10:49:42.1199870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1200319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1200894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1201367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1201580Z 
2022-12-01T10:49:42.1201689Z Running tests...
2022-12-01T10:49:42.1202090Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1202606Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1203059Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1203403Z ok (1.573s)
2022-12-01T10:49:42.1203547Z 
2022-12-01T10:49:42.1203812Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1204137Z Ran 1 test in 1.573s
2022-12-01T10:49:42.1204277Z 
2022-12-01T10:49:42.1204369Z OK
2022-12-01T10:49:42.1204502Z 
2022-12-01T10:49:42.1204627Z Generating XML reports...
2022-12-01T10:49:42.1205194Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201104827.xml
2022-12-01T10:49:42.1205870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1206322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1206900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1207371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1207581Z 
2022-12-01T10:49:42.1207689Z Running tests...
2022-12-01T10:49:42.1208092Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1208605Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1209047Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1209384Z ok (1.563s)
2022-12-01T10:49:42.1209528Z 
2022-12-01T10:49:42.1209793Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1210114Z Ran 1 test in 1.563s
2022-12-01T10:49:42.1210260Z 
2022-12-01T10:49:42.1210359Z OK
2022-12-01T10:49:42.1210486Z 
2022-12-01T10:49:42.1210610Z Generating XML reports...
2022-12-01T10:49:42.1211166Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221201104830.xml
2022-12-01T10:49:42.1211829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1212271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1212840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1213309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1213518Z 
2022-12-01T10:49:42.1213626Z Running tests...
2022-12-01T10:49:42.1214024Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1215550Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1216030Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1216450Z ok (1.569s)
2022-12-01T10:49:42.1216597Z 
2022-12-01T10:49:42.1216872Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1217176Z Ran 1 test in 1.569s
2022-12-01T10:49:42.1217336Z 
2022-12-01T10:49:42.1217430Z OK
2022-12-01T10:49:42.1217563Z 
2022-12-01T10:49:42.1217687Z Generating XML reports...
2022-12-01T10:49:42.1218248Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221201104834.xml
2022-12-01T10:49:42.1218913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1219361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1219941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1220479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1220621Z 
2022-12-01T10:49:42.1220734Z Running tests...
2022-12-01T10:49:42.1221138Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1221656Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1222111Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1222459Z ok (1.560s)
2022-12-01T10:49:42.1222602Z 
2022-12-01T10:49:42.1222867Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1223172Z Ran 1 test in 1.561s
2022-12-01T10:49:42.1223332Z 
2022-12-01T10:49:42.1223425Z OK
2022-12-01T10:49:42.1223560Z 
2022-12-01T10:49:42.1223684Z Generating XML reports...
2022-12-01T10:49:42.1224253Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201104837.xml
2022-12-01T10:49:42.1224925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1225379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1225954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1226404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1226634Z 
2022-12-01T10:49:42.1226741Z Running tests...
2022-12-01T10:49:42.1227142Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1227653Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1228102Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1228449Z ok (1.565s)
2022-12-01T10:49:42.1228593Z 
2022-12-01T10:49:42.1228864Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1229172Z Ran 1 test in 1.565s
2022-12-01T10:49:42.1229332Z 
2022-12-01T10:49:42.1229425Z OK
2022-12-01T10:49:42.1229560Z 
2022-12-01T10:49:42.1229684Z Generating XML reports...
2022-12-01T10:49:42.1230245Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201104840.xml
2022-12-01T10:49:42.1230911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1231441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1231933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1232444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1232674Z 
2022-12-01T10:49:42.1232785Z Running tests...
2022-12-01T10:49:42.1233260Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1233794Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1234333Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1234682Z ok (1.553s)
2022-12-01T10:49:42.1234826Z 
2022-12-01T10:49:42.1235095Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1235400Z Ran 1 test in 1.553s
2022-12-01T10:49:42.1235560Z 
2022-12-01T10:49:42.1235653Z OK
2022-12-01T10:49:42.1235786Z 
2022-12-01T10:49:42.1235911Z Generating XML reports...
2022-12-01T10:49:42.1236475Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104844.xml
2022-12-01T10:49:42.1237146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1237598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1238174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1238632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1238861Z 
2022-12-01T10:49:42.1238969Z Running tests...
2022-12-01T10:49:42.1239370Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1239885Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1240338Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1240990Z [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-12-01T10:49:42.1241523Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-12-01T10:49:42.1241861Z ok (1.652s)
2022-12-01T10:49:42.1242010Z 
2022-12-01T10:49:42.1242277Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1242603Z Ran 1 test in 1.652s
2022-12-01T10:49:42.1242768Z 
2022-12-01T10:49:42.1242859Z OK
2022-12-01T10:49:42.1242974Z 
2022-12-01T10:49:42.1243096Z Generating XML reports...
2022-12-01T10:49:42.1243660Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104847.xml
2022-12-01T10:49:42.1244339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1244771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1245346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1245814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1246043Z 
2022-12-01T10:49:42.1246151Z Running tests...
2022-12-01T10:49:42.1246542Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1247059Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1247532Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1247856Z ok (1.555s)
2022-12-01T10:49:42.1248002Z 
2022-12-01T10:49:42.1248270Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1248593Z Ran 1 test in 1.556s
2022-12-01T10:49:42.1248750Z 
2022-12-01T10:49:42.1248841Z OK
2022-12-01T10:49:42.1248958Z 
2022-12-01T10:49:42.1249083Z Generating XML reports...
2022-12-01T10:49:42.1249648Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104850.xml
2022-12-01T10:49:42.1250329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1250822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1251416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1251944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1252176Z 
2022-12-01T10:49:42.1252286Z Running tests...
2022-12-01T10:49:42.1252672Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1253189Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1253676Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1254015Z ok (11.768s)
2022-12-01T10:49:42.1254163Z 
2022-12-01T10:49:42.1255003Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1255289Z Ran 1 test in 11.768s
2022-12-01T10:49:42.1255449Z 
2022-12-01T10:49:42.1255543Z OK
2022-12-01T10:49:42.1255664Z 
2022-12-01T10:49:42.1255789Z Generating XML reports...
2022-12-01T10:49:42.1256362Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104854.xml
2022-12-01T10:49:42.1257053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1257484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1258056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1258524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1258753Z 
2022-12-01T10:49:42.1258861Z Running tests...
2022-12-01T10:49:42.1259248Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1259764Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1260237Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1260562Z ok (1.574s)
2022-12-01T10:49:42.1260706Z 
2022-12-01T10:49:42.1260976Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1261294Z Ran 1 test in 1.574s
2022-12-01T10:49:42.1261453Z 
2022-12-01T10:49:42.1261546Z OK
2022-12-01T10:49:42.1261660Z 
2022-12-01T10:49:42.1261785Z Generating XML reports...
2022-12-01T10:49:42.1262337Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201104907.xml
2022-12-01T10:49:42.1263010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1263442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1264014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1264486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1264888Z 
2022-12-01T10:49:42.1264962Z Running tests...
2022-12-01T10:49:42.1265211Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1265733Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1266212Z   test_url_with_node_params (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1266541Z ok (1.571s)
2022-12-01T10:49:42.1266686Z 
2022-12-01T10:49:42.1266949Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1267272Z Ran 1 test in 1.571s
2022-12-01T10:49:42.1267433Z 
2022-12-01T10:49:42.1267529Z OK
2022-12-01T10:49:42.1267643Z 
2022-12-01T10:49:42.1267765Z Generating XML reports...
2022-12-01T10:49:42.1268315Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201104911.xml
2022-12-01T10:49:42.1269085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1269536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1270194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1270667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1270899Z 
2022-12-01T10:49:42.1271009Z Running tests...
2022-12-01T10:49:42.1271393Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1271911Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1272384Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1272987Z [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:44719 (errno: 98 - Address already in use).
2022-12-01T10:49:42.1273589Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:44719 (errno: 98 - Address already in use).
2022-12-01T10:49:42.1274056Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-12-01T10:49:42.1274394Z ok (1.565s)
2022-12-01T10:49:42.1274523Z 
2022-12-01T10:49:42.1274793Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1275118Z Ran 1 test in 1.565s
2022-12-01T10:49:42.1275277Z 
2022-12-01T10:49:42.1275370Z OK
2022-12-01T10:49:42.1275503Z 
2022-12-01T10:49:42.1275608Z Generating XML reports...
2022-12-01T10:49:42.1276151Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104914.xml
2022-12-01T10:49:42.1276823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1277269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1277830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1278295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1278527Z 
2022-12-01T10:49:42.1278635Z Running tests...
2022-12-01T10:49:42.1279018Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1279536Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1279996Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1280331Z ok (1.562s)
2022-12-01T10:49:42.1280460Z 
2022-12-01T10:49:42.1280724Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1281043Z Ran 1 test in 1.562s
2022-12-01T10:49:42.1281200Z 
2022-12-01T10:49:42.1281293Z OK
2022-12-01T10:49:42.1281424Z 
2022-12-01T10:49:42.1281528Z Generating XML reports...
2022-12-01T10:49:42.1282079Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104917.xml
2022-12-01T10:49:42.1282749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1283203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1283759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1284230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1284458Z 
2022-12-01T10:49:42.1284566Z Running tests...
2022-12-01T10:49:42.1284950Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1285470Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1285957Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1286527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:49:42.1287187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:49:42.1287644Z ok (1.617s)
2022-12-01T10:49:42.1287793Z 
2022-12-01T10:49:42.1288062Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1288370Z Ran 1 test in 1.617s
2022-12-01T10:49:42.1288532Z 
2022-12-01T10:49:42.1288627Z OK
2022-12-01T10:49:42.1288759Z 
2022-12-01T10:49:42.1288884Z Generating XML reports...
2022-12-01T10:49:42.1289432Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104921.xml
2022-12-01T10:49:42.1290090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1290541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1291126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1291582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1291811Z 
2022-12-01T10:49:42.1291919Z Running tests...
2022-12-01T10:49:42.1292321Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1292837Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1293311Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1293666Z ok (1.566s)
2022-12-01T10:49:42.1293822Z 
2022-12-01T10:49:42.1294087Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1295194Z Ran 1 test in 1.566s
2022-12-01T10:49:42.1295338Z 
2022-12-01T10:49:42.1295432Z OK
2022-12-01T10:49:42.1295546Z 
2022-12-01T10:49:42.1295676Z Generating XML reports...
2022-12-01T10:49:42.1296238Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104924.xml
2022-12-01T10:49:42.1296911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1297344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1297913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1298381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1298610Z 
2022-12-01T10:49:42.1298719Z Running tests...
2022-12-01T10:49:42.1299103Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1299620Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1300119Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1300466Z ok (1.558s)
2022-12-01T10:49:42.1300611Z 
2022-12-01T10:49:42.1300874Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1301199Z Ran 1 test in 1.558s
2022-12-01T10:49:42.1301357Z 
2022-12-01T10:49:42.1301449Z OK
2022-12-01T10:49:42.1301564Z 
2022-12-01T10:49:42.1301686Z Generating XML reports...
2022-12-01T10:49:42.1302228Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104927.xml
2022-12-01T10:49:42.1302897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1303331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1303905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1304470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1304715Z 
2022-12-01T10:49:42.1304825Z Running tests...
2022-12-01T10:49:42.1305217Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1305819Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1306285Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1306606Z ok (1.573s)
2022-12-01T10:49:42.1306752Z 
2022-12-01T10:49:42.1307015Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1307336Z Ran 1 test in 1.573s
2022-12-01T10:49:42.1307494Z 
2022-12-01T10:49:42.1307568Z OK
2022-12-01T10:49:42.1307700Z 
2022-12-01T10:49:42.1307822Z Generating XML reports...
2022-12-01T10:49:42.1308365Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104931.xml
2022-12-01T10:49:42.1309038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1309470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1310042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1310520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1310749Z 
2022-12-01T10:49:42.1310858Z Running tests...
2022-12-01T10:49:42.1311244Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1311758Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1312222Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1312541Z ok (3.567s)
2022-12-01T10:49:42.1312685Z 
2022-12-01T10:49:42.1312949Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1313268Z Ran 1 test in 3.567s
2022-12-01T10:49:42.1313431Z 
2022-12-01T10:49:42.1313506Z OK
2022-12-01T10:49:42.1313638Z 
2022-12-01T10:49:42.1313760Z Generating XML reports...
2022-12-01T10:49:42.1314304Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104934.xml
2022-12-01T10:49:42.1314979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:49:42.1315409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:49:42.1316068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:49:42.1316449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:49:42.1316679Z 
2022-12-01T10:49:42.1316769Z Running tests...
2022-12-01T10:49:42.1317168Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1317685Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-12-01T10:49:42.1318139Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:49:42.1318446Z ok (1.558s)
2022-12-01T10:49:42.1318596Z 
2022-12-01T10:49:42.1318861Z ----------------------------------------------------------------------
2022-12-01T10:49:42.1319181Z Ran 1 test in 1.558s
2022-12-01T10:49:42.1319340Z 
2022-12-01T10:49:42.1319413Z OK
2022-12-01T10:49:42.1319545Z 
2022-12-01T10:49:42.1319668Z Generating XML reports...
2022-12-01T10:49:42.1320212Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104940.xml
2022-12-01T10:49:42.1320524Z 
2022-12-01T10:49:42.1320910Z ##[endgroup]
2022-12-01T10:49:42.1321464Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_zse1uhl2)
2022-12-01T10:49:42.1321786Z 
2022-12-01T10:49:42.1322051Z Running distributed/fsdp/test_fsdp_comm ... [2022-12-01 10:49:42.112741]
2022-12-01T10:49:42.1322794Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:49:42.113015]
2022-12-01T10:50:18.6899170Z 
2022-12-01T10:50:18.6899709Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm
2022-12-01T10:50:18.6900656Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_cwemw55y)
2022-12-01T10:50:18.6901012Z 
2022-12-01T10:50:18.6901127Z Running tests...
2022-12-01T10:50:18.6901631Z ----------------------------------------------------------------------
2022-12-01T10:50:18.6902183Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2022-12-01T10:50:18.6905097Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:50:18.6906443Z Tests FSDP's communication cost in terms of calls to collective ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:50:18.6907101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32308
2022-12-01T10:50:18.6907593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32309
2022-12-01T10:50:18.6908136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6908583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6909665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6910468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6911054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6911505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6912121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6912580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6913282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.6913917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.6915154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6916394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6917541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.6918004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.6918440Z dist init r=0, world=2
2022-12-01T10:50:18.6918694Z dist init r=1, world=2
2022-12-01T10:50:18.6918933Z ok (5.911s)
2022-12-01T10:50:18.6919350Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:50:18.6920101Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32391
2022-12-01T10:50:18.6920634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32392
2022-12-01T10:50:18.6921249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6921682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6922262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6922993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6923597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6924154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6924732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6925205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6925644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.6926143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.6926803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6927499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6928007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.6928483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.6928845Z dist init r=0, world=2
2022-12-01T10:50:18.6929078Z dist init r=1, world=2
2022-12-01T10:50:18.6929318Z ok (4.313s)
2022-12-01T10:50:18.6929699Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:50:18.6930407Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32474
2022-12-01T10:50:18.6930917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32475
2022-12-01T10:50:18.6931534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6931986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6932543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6933021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6933597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6934038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6935257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6936126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6937025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.6937927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.6938652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6939348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6939875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.6940330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.6940680Z dist init r=1, world=2
2022-12-01T10:50:18.6940937Z dist init r=0, world=2
2022-12-01T10:50:18.6941159Z ok (4.313s)
2022-12-01T10:50:18.6941589Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:50:18.6942454Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32557
2022-12-01T10:50:18.6943008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32558
2022-12-01T10:50:18.6943682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6944127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6944701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6945173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6945730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6946173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6946758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6947208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6947673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.6948171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.6948829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6949504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6950028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.6950499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.6950860Z dist init r=0, world=2
2022-12-01T10:50:18.6951100Z dist init r=1, world=2
2022-12-01T10:50:18.6951341Z ok (4.413s)
2022-12-01T10:50:18.6951726Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:50:18.6952417Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32640
2022-12-01T10:50:18.6952951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32641
2022-12-01T10:50:18.6953560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6953992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6954569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6955040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6955622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6956051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6956627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6957092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6957551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.6958036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.6958694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6959383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6959956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.6960438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.6961711Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.6962540Z   warnings.warn(
2022-12-01T10:50:18.6963662Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.6964418Z   warnings.warn(
2022-12-01T10:50:18.6964649Z dist init r=0, world=2
2022-12-01T10:50:18.6964899Z dist init r=1, world=2
2022-12-01T10:50:18.6965137Z ok (3.912s)
2022-12-01T10:50:18.6965546Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:50:18.6966287Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32723
2022-12-01T10:50:18.6966819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32724
2022-12-01T10:50:18.6967428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6967864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6968441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6968920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6969500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6969924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6970498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6970966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6971405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.6971904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.6972564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6973261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6973764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.6974235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.6975906Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.6976778Z   warnings.warn(
2022-12-01T10:50:18.6977991Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.6978868Z   warnings.warn(
2022-12-01T10:50:18.6979131Z dist init r=1, world=2
2022-12-01T10:50:18.6979394Z dist init r=0, world=2
2022-12-01T10:50:18.6979625Z ok (4.012s)
2022-12-01T10:50:18.6980027Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2022-12-01T10:50:18.6980771Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32806
2022-12-01T10:50:18.6981336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32807
2022-12-01T10:50:18.6981970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6982452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6983064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6983545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6984162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6984640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6985249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6985734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6986216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.6986746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.6987442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6988151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.6988704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.6989205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.6990505Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.6991288Z   warnings.warn(
2022-12-01T10:50:18.6992458Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.6993261Z   warnings.warn(
2022-12-01T10:50:18.6993518Z dist init r=0, world=2
2022-12-01T10:50:18.6993762Z dist init r=1, world=2
2022-12-01T10:50:18.6994009Z ok (4.012s)
2022-12-01T10:50:18.6994520Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2022-12-01T10:50:18.6995370Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32889
2022-12-01T10:50:18.6995922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32890
2022-12-01T10:50:18.6996573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6997054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6997647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.6998145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.6998764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:18.6999242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:18.6999832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:18.7000335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:18.7000817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:18.7001325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:18.7002020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.7002750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:18.7003307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:18.7003790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:18.7005088Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.7005891Z   warnings.warn(
2022-12-01T10:50:18.7007069Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:18.7007874Z   warnings.warn(
2022-12-01T10:50:18.7008120Z dist init r=0, world=2
2022-12-01T10:50:18.7008380Z dist init r=1, world=2
2022-12-01T10:50:18.7008629Z ok (3.913s)
2022-12-01T10:50:18.7008783Z 
2022-12-01T10:50:18.7009054Z ----------------------------------------------------------------------
2022-12-01T10:50:18.7009403Z Ran 8 tests in 34.800s
2022-12-01T10:50:18.7009574Z 
2022-12-01T10:50:18.7009669Z OK
2022-12-01T10:50:18.7009807Z 
2022-12-01T10:50:18.7009936Z Generating XML reports...
2022-12-01T10:50:18.7010539Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221201104943.xml
2022-12-01T10:50:18.7010905Z 
2022-12-01T10:50:18.7011318Z ##[endgroup]
2022-12-01T10:50:18.7012021Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_cwemw55y)
2022-12-01T10:50:18.7012405Z 
2022-12-01T10:50:18.7012706Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler ... [2022-12-01 10:50:18.690205]
2022-12-01T10:50:18.7013541Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:50:18.690507]
2022-12-01T10:50:54.6524926Z 
2022-12-01T10:50:54.6526037Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_sharded_grad_scaler
2022-12-01T10:50:54.6527912Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_oznb93ud)
2022-12-01T10:50:54.6528699Z 
2022-12-01T10:50:54.6528950Z Running tests...
2022-12-01T10:50:54.6529996Z ----------------------------------------------------------------------
2022-12-01T10:50:54.6530709Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler
2022-12-01T10:50:54.6531395Z   test_grad_scaling (__main__.TestShardGradScaler) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:50:54.6531853Z ok (1.597s)
2022-12-01T10:50:54.6532299Z   test_inf_gradients_skip_optim_step (__main__.TestShardGradScaler) ... ok (0.002s)
2022-12-01T10:50:54.6532998Z   test_scaling_unscaling_sparse (__main__.TestShardGradScaler) ... ok (0.006s)
2022-12-01T10:50:54.6533698Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33022
2022-12-01T10:50:54.6534651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33023
2022-12-01T10:50:54.6535405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6536071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6536848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6537388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6538092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6538611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6539270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6539834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6540369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6541302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6542874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6544558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6545638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6546596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6547624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6565487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6567768Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6569500Z   warnings.warn(
2022-12-01T10:50:54.6571230Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6571993Z   warnings.warn(
2022-12-01T10:50:54.6572759Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6573305Z   warnings.warn(
2022-12-01T10:50:54.6574049Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6575500Z   warnings.warn(
2022-12-01T10:50:54.6576420Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6576974Z   warnings.warn(
2022-12-01T10:50:54.6577737Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6578255Z   warnings.warn(
2022-12-01T10:50:54.6579172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6579839Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6580771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6581409Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6581817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6582312Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6582678Z dist init r=0, world=2
2022-12-01T10:50:54.6582916Z dist init r=1, world=2
2022-12-01T10:50:54.6583156Z ok (4.013s)
2022-12-01T10:50:54.6583681Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33105
2022-12-01T10:50:54.6584282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33106
2022-12-01T10:50:54.6584902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6585355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6585935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6586391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6587087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6587557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6588117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6588665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6589126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6589632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6590276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6590966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6591501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6591980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6592447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6592935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6594169Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6594929Z   warnings.warn(
2022-12-01T10:50:54.6596017Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6596759Z   warnings.warn(
2022-12-01T10:50:54.6597520Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6598057Z   warnings.warn(
2022-12-01T10:50:54.6598785Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6599324Z   warnings.warn(
2022-12-01T10:50:54.6600094Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6600651Z   warnings.warn(
2022-12-01T10:50:54.6601388Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6601926Z   warnings.warn(
2022-12-01T10:50:54.6602829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6603492Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6604481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6605224Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6605634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6606123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6606467Z dist init r=0, world=2
2022-12-01T10:50:54.6606719Z dist init r=1, world=2
2022-12-01T10:50:54.6606963Z ok (4.011s)
2022-12-01T10:50:54.6607513Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33188
2022-12-01T10:50:54.6608139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33189
2022-12-01T10:50:54.6608762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6609222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6609784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6610257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6610840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6611289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6611843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6612313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6612770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6613275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6613918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6615119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6615658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6616113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6616596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6617090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6618328Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6619091Z   warnings.warn(
2022-12-01T10:50:54.6620278Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6621114Z   warnings.warn(
2022-12-01T10:50:54.6621927Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6622599Z   warnings.warn(
2022-12-01T10:50:54.6623386Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6623961Z   warnings.warn(
2022-12-01T10:50:54.6624775Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6625356Z   warnings.warn(
2022-12-01T10:50:54.6626156Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6626748Z   warnings.warn(
2022-12-01T10:50:54.6627704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6628407Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6629389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6630078Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6630500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6631015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6631379Z dist init r=0, world=2
2022-12-01T10:50:54.6631647Z dist init r=1, world=2
2022-12-01T10:50:54.6631900Z ok (4.011s)
2022-12-01T10:50:54.6632464Z   test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33271
2022-12-01T10:50:54.6633115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33272
2022-12-01T10:50:54.6633766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6634246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6634840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6635343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6635961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6636439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6637030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6637548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6638004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6638531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6639274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6640020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6640636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6641167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6641679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6642195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6643509Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6644326Z   warnings.warn(
2022-12-01T10:50:54.6645480Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6646284Z   warnings.warn(
2022-12-01T10:50:54.6647085Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6647665Z   warnings.warn(
2022-12-01T10:50:54.6648444Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6649019Z   warnings.warn(
2022-12-01T10:50:54.6649832Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6650420Z   warnings.warn(
2022-12-01T10:50:54.6651217Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6651799Z   warnings.warn(
2022-12-01T10:50:54.6652760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6653466Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6654937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6655589Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6656000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6656493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6656935Z dist init r=0, world=2
2022-12-01T10:50:54.6657204Z dist init r=1, world=2
2022-12-01T10:50:54.6657447Z ok (4.011s)
2022-12-01T10:50:54.6657969Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33354
2022-12-01T10:50:54.6658715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33355
2022-12-01T10:50:54.6659347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6659803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6660364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6660837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6661420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6661869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6662430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6662893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6663349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6663831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6664492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6665181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6665708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6666166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6666643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6667134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6668361Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6669119Z   warnings.warn(
2022-12-01T10:50:54.6670203Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6670956Z   warnings.warn(
2022-12-01T10:50:54.6671331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6671821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6672806Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6674111Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6675303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6675871Z   warnings.warn(
2022-12-01T10:50:54.6676610Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6677122Z   warnings.warn(
2022-12-01T10:50:54.6677894Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6678447Z   warnings.warn(
2022-12-01T10:50:54.6679210Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6679730Z   warnings.warn(
2022-12-01T10:50:54.6680631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6681293Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6682224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6682864Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6683272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6683756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6684121Z dist init r=1, world=2
2022-12-01T10:50:54.6684357Z dist init r=0, world=2
2022-12-01T10:50:54.6684598Z ok (4.111s)
2022-12-01T10:50:54.6685124Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33437
2022-12-01T10:50:54.6685716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33438
2022-12-01T10:50:54.6686336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6686797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6687376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6687830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6688408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6688857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6689406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6689867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6690388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6690907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6691625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6692315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6692841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6693312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6693776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6694262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6696169Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6696938Z   warnings.warn(
2022-12-01T10:50:54.6698024Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6698773Z   warnings.warn(
2022-12-01T10:50:54.6699154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6699641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6700632Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6701864Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6702978Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6703524Z   warnings.warn(
2022-12-01T10:50:54.6704286Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6704824Z   warnings.warn(
2022-12-01T10:50:54.6705573Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6706122Z   warnings.warn(
2022-12-01T10:50:54.6706971Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6707524Z   warnings.warn(
2022-12-01T10:50:54.6708418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6709158Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6710080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6710741Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6711131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6711627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6711996Z dist init r=1, world=2
2022-12-01T10:50:54.6712234Z dist init r=0, world=2
2022-12-01T10:50:54.6712477Z ok (4.111s)
2022-12-01T10:50:54.6713023Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33520
2022-12-01T10:50:54.6713657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33521
2022-12-01T10:50:54.6714253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6714704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6715281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6715742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6716329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6716777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6717351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6717801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6718260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6718758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6719411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6720089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6720612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6721087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6721547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6722030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6723252Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6724067Z   warnings.warn(
2022-12-01T10:50:54.6725182Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6725967Z   warnings.warn(
2022-12-01T10:50:54.6726343Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6726824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6727832Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6729073Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6730177Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6730717Z   warnings.warn(
2022-12-01T10:50:54.6731467Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6732009Z   warnings.warn(
2022-12-01T10:50:54.6732751Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6733301Z   warnings.warn(
2022-12-01T10:50:54.6734052Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6735131Z   warnings.warn(
2022-12-01T10:50:54.6736030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6736701Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6737623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6738288Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6738677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6739166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6739531Z dist init r=0, world=2
2022-12-01T10:50:54.6739783Z dist init r=1, world=2
2022-12-01T10:50:54.6740006Z ok (4.111s)
2022-12-01T10:50:54.6740638Z   test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none (__main__.TestShardedGradScalerParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33603
2022-12-01T10:50:54.6741334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33604
2022-12-01T10:50:54.6742017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6742472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6743048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6743518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6744077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:50:54.6744527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:50:54.6745099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:50:54.6745566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:50:54.6746003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:50:54.6746511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:50:54.6747168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6747846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:50:54.6748370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:50:54.6748842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:50:54.6749325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6749795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6751021Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6751781Z   warnings.warn(
2022-12-01T10:50:54.6752889Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:50:54.6753629Z   warnings.warn(
2022-12-01T10:50:54.6753983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6754473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6755481Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6756718Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:50:54.6757904Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6758501Z   warnings.warn(
2022-12-01T10:50:54.6759252Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:50:54.6759794Z   warnings.warn(
2022-12-01T10:50:54.6760558Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6761087Z   warnings.warn(
2022-12-01T10:50:54.6761841Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:50:54.6762387Z   warnings.warn(
2022-12-01T10:50:54.6763292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6763936Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6764856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_deprecated.py:35: FutureWarning: torch.testing.assert_allclose() is deprecated since 1.12 and will be removed in 1.14. Use torch.testing.assert_close() instead. For detailed upgrade instructions see https://github.com/pytorch/pytorch/issues/61844.
2022-12-01T10:50:54.6765510Z   warnings.warn(msg, FutureWarning)
2022-12-01T10:50:54.6765920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6766389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:50:54.6766761Z dist init r=0, world=2
2022-12-01T10:50:54.6767013Z dist init r=1, world=2
2022-12-01T10:50:54.6767235Z ok (4.111s)
2022-12-01T10:50:54.6767384Z 
2022-12-01T10:50:54.6767662Z ----------------------------------------------------------------------
2022-12-01T10:50:54.6767997Z Ran 11 tests in 34.097s
2022-12-01T10:50:54.6768159Z 
2022-12-01T10:50:54.6768256Z OK
2022-12-01T10:50:54.6768370Z 
2022-12-01T10:50:54.6768496Z Generating XML reports...
2022-12-01T10:50:54.6769129Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221201105020.xml
2022-12-01T10:50:54.6770019Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221201105020.xml
2022-12-01T10:50:54.6770442Z 
2022-12-01T10:50:54.6770808Z ##[endgroup]
2022-12-01T10:50:54.6771466Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_sharded_grad_scaler (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_sharded_grad_scaler_oznb93ud)
2022-12-01T10:50:54.6771860Z 
2022-12-01T10:50:54.6772117Z Running distributed/test_c10d_common ... [2022-12-01 10:50:54.652975]
2022-12-01T10:50:54.6772811Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:50:54.653234]
2022-12-01T10:51:51.5190849Z 
2022-12-01T10:51:51.5191329Z Expand the folded group to see the log file of distributed/test_c10d_common
2022-12-01T10:51:51.5193814Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_fxsbnnbu)
2022-12-01T10:51:51.5194647Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_debug_level>]>
2022-12-01T10:51:51.5195071Z test_debug_level (__main__.CommTest)
2022-12-01T10:51:51.5195878Z <unittest.suite.TestSuite tests=[<__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]>
2022-12-01T10:51:51.5196908Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:51:51.5197355Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:51:51.5197811Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:51:51.5199111Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2022-12-01T10:51:51.5199957Z <unittest.suite.TestSuite tests=[<__main__.PythonProcessGroupExtensionTest testMethod=test_backend_class_attr>, <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]>
2022-12-01T10:51:51.5200844Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:51:51.5201302Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:51:51.5201758Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:51:51.5204489Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2022-12-01T10:51:51.5205003Z <unittest.suite.TestSuite tests=[<__main__.ReduceOpTest testMethod=test_op_isinstance_of_reduceop>]>
2022-12-01T10:51:51.5205485Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest)
2022-12-01T10:51:51.5206189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5206654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5207257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5207742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5207954Z 
2022-12-01T10:51:51.5208101Z Running tests...
2022-12-01T10:51:51.5208462Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5209041Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5209657Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5210103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33754
2022-12-01T10:51:51.5210501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33755
2022-12-01T10:51:51.5211107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5211564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5212122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5212603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5213187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5213632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5214186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5215641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5216088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:51.5216711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:51.5217107Z ok (3.761s)
2022-12-01T10:51:51.5217265Z 
2022-12-01T10:51:51.5217666Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5218020Z Ran 1 test in 3.762s
2022-12-01T10:51:51.5218169Z 
2022-12-01T10:51:51.5218264Z OK
2022-12-01T10:51:51.5218404Z 
2022-12-01T10:51:51.5218530Z Generating XML reports...
2022-12-01T10:51:51.5219110Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221201105058.xml
2022-12-01T10:51:51.5219798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5220278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5220886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5221394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5221641Z 
2022-12-01T10:51:51.5221735Z Running tests...
2022-12-01T10:51:51.5222160Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5222729Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5223261Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5223654Z ok (1.543s)
2022-12-01T10:51:51.5223807Z 
2022-12-01T10:51:51.5224091Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5224436Z Ran 1 test in 1.544s
2022-12-01T10:51:51.5224584Z 
2022-12-01T10:51:51.5224678Z OK
2022-12-01T10:51:51.5224817Z 
2022-12-01T10:51:51.5224946Z Generating XML reports...
2022-12-01T10:51:51.5225597Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105104.xml
2022-12-01T10:51:51.5226347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5226836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5227446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5227943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5228188Z 
2022-12-01T10:51:51.5228281Z Running tests...
2022-12-01T10:51:51.5228701Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5229269Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5229802Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5230202Z ok (1.558s)
2022-12-01T10:51:51.5230355Z 
2022-12-01T10:51:51.5230641Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5230987Z Ran 1 test in 1.558s
2022-12-01T10:51:51.5231136Z 
2022-12-01T10:51:51.5231233Z OK
2022-12-01T10:51:51.5231372Z 
2022-12-01T10:51:51.5231498Z Generating XML reports...
2022-12-01T10:51:51.5232147Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105108.xml
2022-12-01T10:51:51.5232889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5233368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5233974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5234465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5234706Z 
2022-12-01T10:51:51.5234798Z Running tests...
2022-12-01T10:51:51.5235290Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5235872Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5236470Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5236873Z ok (1.543s)
2022-12-01T10:51:51.5237026Z 
2022-12-01T10:51:51.5237313Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5237656Z Ran 1 test in 1.543s
2022-12-01T10:51:51.5237805Z 
2022-12-01T10:51:51.5237898Z OK
2022-12-01T10:51:51.5238036Z 
2022-12-01T10:51:51.5238166Z Generating XML reports...
2022-12-01T10:51:51.5238818Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105111.xml
2022-12-01T10:51:51.5239565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5240045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5240654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5241158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5241401Z 
2022-12-01T10:51:51.5241494Z Running tests...
2022-12-01T10:51:51.5241917Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5242479Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5243033Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5243418Z ok (1.547s)
2022-12-01T10:51:51.5243572Z 
2022-12-01T10:51:51.5243854Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5244199Z Ran 1 test in 1.547s
2022-12-01T10:51:51.5244348Z 
2022-12-01T10:51:51.5244446Z OK
2022-12-01T10:51:51.5244584Z 
2022-12-01T10:51:51.5244709Z Generating XML reports...
2022-12-01T10:51:51.5245357Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105115.xml
2022-12-01T10:51:51.5246104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5246581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5247188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5247686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5247983Z 
2022-12-01T10:51:51.5248078Z Running tests...
2022-12-01T10:51:51.5248503Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5249068Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5249628Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5250148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33993
2022-12-01T10:51:51.5250624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33994
2022-12-01T10:51:51.5251102Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 33995
2022-12-01T10:51:51.5251560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 33996
2022-12-01T10:51:51.5252199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5252675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5253284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5253827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5255273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5255855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5256433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5256905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5257487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5257931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5258483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5258955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5259530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5259960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5260535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5260999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5261436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:51.5261893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:51.5262350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:51:51.5262814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:51.5263157Z ok (3.860s)
2022-12-01T10:51:51.5263292Z 
2022-12-01T10:51:51.5263570Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5263897Z Ran 1 test in 3.860s
2022-12-01T10:51:51.5264063Z 
2022-12-01T10:51:51.5264156Z OK
2022-12-01T10:51:51.5264270Z 
2022-12-01T10:51:51.5264395Z Generating XML reports...
2022-12-01T10:51:51.5265034Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105119.xml
2022-12-01T10:51:51.5265774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5266224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5266778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5267246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5267475Z 
2022-12-01T10:51:51.5267587Z Running tests...
2022-12-01T10:51:51.5267973Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5268503Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5269029Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5269530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34164
2022-12-01T10:51:51.5269958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34165
2022-12-01T10:51:51.5270389Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34166
2022-12-01T10:51:51.5270824Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34167
2022-12-01T10:51:51.5271412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5271935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5272528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5273056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5273615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5274053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5274624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5275070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5275645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5276087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5276659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5277109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5277686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5278124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5278693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5279134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5279567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:51.5280207Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-12-01T10:51:51.5280697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:51:51.5281326Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-12-01T10:51:51.5281829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:51.5282298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:51.5282768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:51:51.5283257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:51:51.5283745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:51:51.5284231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:51:51.5284872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5285567Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5286253Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5286910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5287297Z ok (6.659s)
2022-12-01T10:51:51.5287447Z 
2022-12-01T10:51:51.5287714Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5288041Z Ran 1 test in 6.659s
2022-12-01T10:51:51.5288181Z 
2022-12-01T10:51:51.5288273Z OK
2022-12-01T10:51:51.5288404Z 
2022-12-01T10:51:51.5288526Z Generating XML reports...
2022-12-01T10:51:51.5289221Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105125.xml
2022-12-01T10:51:51.5289957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5290462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5291036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5291507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5291735Z 
2022-12-01T10:51:51.5291825Z Running tests...
2022-12-01T10:51:51.5292225Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5292754Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5293280Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5293769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34344
2022-12-01T10:51:51.5294218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34345
2022-12-01T10:51:51.5295350Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34346
2022-12-01T10:51:51.5295775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34347
2022-12-01T10:51:51.5296391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5296842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5297419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5297871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5298455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5298900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5299459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5299922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5300497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5300939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5301487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5301950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5302528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5302968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5303516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5303979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5304411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:51.5304863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:51.5305324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:51:51.5305790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:51.5306133Z ok (3.850s)
2022-12-01T10:51:51.5306262Z 
2022-12-01T10:51:51.5306536Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5306979Z Ran 1 test in 3.850s
2022-12-01T10:51:51.5307154Z 
2022-12-01T10:51:51.5307247Z OK
2022-12-01T10:51:51.5307378Z 
2022-12-01T10:51:51.5307484Z Generating XML reports...
2022-12-01T10:51:51.5308211Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105134.xml
2022-12-01T10:51:51.5308947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5309394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5309943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5310408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5310635Z 
2022-12-01T10:51:51.5310743Z Running tests...
2022-12-01T10:51:51.5311125Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5311660Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5312174Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5312673Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34515
2022-12-01T10:51:51.5313099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34516
2022-12-01T10:51:51.5313531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 34517
2022-12-01T10:51:51.5313970Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 34518
2022-12-01T10:51:51.5314558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5315008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5315584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5316050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5316607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5317051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5317621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5318064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5318634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5319073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5319643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5320095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5320667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5321109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5321676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5322117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5322554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:51:51.5323193Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-12-01T10:51:51.5323678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:51:51.5324384Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2022-12-01T10:51:51.5324895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:51:51.5325417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:51:51.5325886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:51:51.5326379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-12-01T10:51:51.5326862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-12-01T10:51:51.5327345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:51:51.5327987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5328679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5329371Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5330046Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-12-01T10:51:51.5330414Z ok (6.566s)
2022-12-01T10:51:51.5330563Z 
2022-12-01T10:51:51.5330833Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5331160Z Ran 1 test in 6.566s
2022-12-01T10:51:51.5331321Z 
2022-12-01T10:51:51.5331394Z OK
2022-12-01T10:51:51.5331527Z 
2022-12-01T10:51:51.5331650Z Generating XML reports...
2022-12-01T10:51:51.5332289Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105140.xml
2022-12-01T10:51:51.5333013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:51:51.5333460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:51:51.5334040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:51:51.5335093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:51:51.5335340Z 
2022-12-01T10:51:51.5335435Z Running tests...
2022-12-01T10:51:51.5335845Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5336381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2022-12-01T10:51:51.5336870Z   test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:51:51.5337208Z ok (1.518s)
2022-12-01T10:51:51.5337353Z 
2022-12-01T10:51:51.5337622Z ----------------------------------------------------------------------
2022-12-01T10:51:51.5337946Z Ran 1 test in 1.519s
2022-12-01T10:51:51.5338122Z 
2022-12-01T10:51:51.5338197Z OK
2022-12-01T10:51:51.5338330Z 
2022-12-01T10:51:51.5338460Z Generating XML reports...
2022-12-01T10:51:51.5339017Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221201105149.xml
2022-12-01T10:51:51.5339343Z 
2022-12-01T10:51:51.5342023Z ##[endgroup]
2022-12-01T10:51:51.5342621Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_fxsbnnbu)
2022-12-01T10:51:51.5342960Z 
2022-12-01T10:51:51.5343230Z Running distributed/fsdp/test_fsdp_meta ... [2022-12-01 10:51:51.519328]
2022-12-01T10:51:51.5343902Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:51:51.519594]
2022-12-01T10:52:22.1720711Z 
2022-12-01T10:52:22.1723722Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta
2022-12-01T10:52:22.1725036Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_7584cnds)
2022-12-01T10:52:22.1725803Z 
2022-12-01T10:52:22.1725924Z Running tests...
2022-12-01T10:52:22.1726529Z ----------------------------------------------------------------------
2022-12-01T10:52:22.1727120Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2022-12-01T10:52:22.1727676Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:52:22.1728344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34729
2022-12-01T10:52:22.1728809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34730
2022-12-01T10:52:22.1734192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1735233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1735820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1736346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1737416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1738005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1739253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1740099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1740886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:22.1741749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:22.1742920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1743638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1744145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:22.1744623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:22.1744978Z dist init r=0, world=2
2022-12-01T10:52:22.1745232Z dist init r=1, world=2
2022-12-01T10:52:22.1745454Z ok (4.983s)
2022-12-01T10:52:22.1745924Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-12-01T10:52:22.1746617Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34808
2022-12-01T10:52:22.1747182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34809
2022-12-01T10:52:22.1747803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1748257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1748837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1749293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1749872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1750320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1751090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1751569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1752179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:22.1752676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:22.1753326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1754016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1754543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:22.1755020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:22.1755879Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1756503Z   warnings.warn(
2022-12-01T10:52:22.1757254Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1757783Z   warnings.warn(
2022-12-01T10:52:22.1758550Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1759075Z   warnings.warn(
2022-12-01T10:52:22.1759840Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1760394Z   warnings.warn(
2022-12-01T10:52:22.1760647Z dist init r=0, world=2
2022-12-01T10:52:22.1760882Z dist init r=1, world=2
2022-12-01T10:52:22.1761125Z ok (4.010s)
2022-12-01T10:52:22.1761614Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34891
2022-12-01T10:52:22.1762175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34892
2022-12-01T10:52:22.1762792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1763246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1763833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1764380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1764963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1765400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1765979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1766448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1766886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:22.1767385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:22.1768048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1768818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1769383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:22.1769854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:22.1770725Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1771276Z   warnings.warn(
2022-12-01T10:52:22.1772011Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1772546Z   warnings.warn(
2022-12-01T10:52:22.1773317Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1773865Z   warnings.warn(
2022-12-01T10:52:22.1774899Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1775463Z   warnings.warn(
2022-12-01T10:52:22.1775715Z dist init r=1, world=2
2022-12-01T10:52:22.1775955Z dist init r=0, world=2
2022-12-01T10:52:22.1776196Z ok (4.010s)
2022-12-01T10:52:22.1776691Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34974
2022-12-01T10:52:22.1777283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34975
2022-12-01T10:52:22.1777889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1778346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1778928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1779386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1779968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1780412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1780988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1781441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1781896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:22.1782401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:22.1783063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1783737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1784266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:22.1784741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:22.1785691Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1786262Z   warnings.warn(
2022-12-01T10:52:22.1787018Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1787642Z   warnings.warn(
2022-12-01T10:52:22.1788390Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1788935Z   warnings.warn(
2022-12-01T10:52:22.1789690Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1790242Z   warnings.warn(
2022-12-01T10:52:22.1790472Z dist init r=0, world=2
2022-12-01T10:52:22.1790726Z dist init r=1, world=2
2022-12-01T10:52:22.1790970Z ok (3.910s)
2022-12-01T10:52:22.1791441Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35057
2022-12-01T10:52:22.1792022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35058
2022-12-01T10:52:22.1792631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1793083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1793645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1794116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1794703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1795151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1795709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1796174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1796631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:22.1797110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:22.1797778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1798474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1799000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:22.1799455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:22.1800322Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1800865Z   warnings.warn(
2022-12-01T10:52:22.1801611Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1802132Z   warnings.warn(
2022-12-01T10:52:22.1802960Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1803522Z   warnings.warn(
2022-12-01T10:52:22.1804332Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1804851Z   warnings.warn(
2022-12-01T10:52:22.1805101Z dist init r=0, world=2
2022-12-01T10:52:22.1805356Z dist init r=1, world=2
2022-12-01T10:52:22.1805578Z ok (4.011s)
2022-12-01T10:52:22.1806086Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-12-01T10:52:22.1806822Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-12-01T10:52:22.1807544Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-12-01T10:52:22.1808232Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-12-01T10:52:22.1808936Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35140
2022-12-01T10:52:22.1809498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35141
2022-12-01T10:52:22.1810111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1810548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1811132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1811608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1812196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1812623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1813203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1813674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1814134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:22.1815073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:22.1815745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1816436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1816947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:22.1817419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:22.1818282Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1818828Z   warnings.warn(
2022-12-01T10:52:22.1819652Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1820209Z   warnings.warn(
2022-12-01T10:52:22.1820977Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1821603Z   warnings.warn(
2022-12-01T10:52:22.1822351Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1822888Z   warnings.warn(
2022-12-01T10:52:22.1823136Z dist init r=0, world=2
2022-12-01T10:52:22.1823389Z dist init r=1, world=2
2022-12-01T10:52:22.1823609Z ok (3.910s)
2022-12-01T10:52:22.1824087Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35223
2022-12-01T10:52:22.1824655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35224
2022-12-01T10:52:22.1825257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1825712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1826292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1826768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1827326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:22.1827776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:22.1828351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:22.1828803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:22.1829260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:22.1829765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:22.1830426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1831098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:22.1831622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:22.1832096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:22.1832972Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1833512Z   warnings.warn(
2022-12-01T10:52:22.1834269Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:52:22.1834807Z   warnings.warn(
2022-12-01T10:52:22.1835574Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1836097Z   warnings.warn(
2022-12-01T10:52:22.1837368Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:52:22.1837933Z   warnings.warn(
2022-12-01T10:52:22.1838187Z dist init r=1, world=2
2022-12-01T10:52:22.1838480Z dist init r=0, world=2
2022-12-01T10:52:22.1838722Z ok (4.010s)
2022-12-01T10:52:22.1839210Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-12-01T10:52:22.1839878Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-12-01T10:52:22.1840238Z 
2022-12-01T10:52:22.1840519Z ----------------------------------------------------------------------
2022-12-01T10:52:22.1840854Z Ran 14 tests in 28.850s
2022-12-01T10:52:22.1841021Z 
2022-12-01T10:52:22.1841130Z OK (skipped=7)
2022-12-01T10:52:22.1841265Z 
2022-12-01T10:52:22.1841390Z Generating XML reports...
2022-12-01T10:52:22.1842015Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221201105152.xml
2022-12-01T10:52:22.1842385Z 
2022-12-01T10:52:22.1842788Z ##[endgroup]
2022-12-01T10:52:22.1843366Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_7584cnds)
2022-12-01T10:52:22.1843715Z 
2022-12-01T10:52:22.1844006Z Running distributed/fsdp/test_fsdp_tp_integration ... [2022-12-01 10:52:22.172151]
2022-12-01T10:52:22.1844719Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:22.172412]
2022-12-01T10:52:42.6776327Z 
2022-12-01T10:52:42.6777609Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration
2022-12-01T10:52:42.6779168Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_1b_s9uij)
2022-12-01T10:52:42.6779766Z 
2022-12-01T10:52:42.6779863Z Running tests...
2022-12-01T10:52:42.6780413Z ----------------------------------------------------------------------
2022-12-01T10:52:42.6781037Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration
2022-12-01T10:52:42.6781501Z   test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
2022-12-01T10:52:42.6781966Z Tests checkpointing for TP + FSDP integration. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:52:42.6782400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35341
2022-12-01T10:52:42.6783268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35342
2022-12-01T10:52:42.6784421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6785116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6786410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6787342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6788471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6789366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6790058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6790621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6790997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:42.6791497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:42.6792462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6793518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6794900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:42.6795767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:42.6796495Z dist init r=0, world=2
2022-12-01T10:52:42.6796850Z dist init r=1, world=2
2022-12-01T10:52:42.6797139Z skip: Need at least 4 CUDA devices (4.999s)
2022-12-01T10:52:42.6797602Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-12-01T10:52:42.6798423Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35420
2022-12-01T10:52:42.6798945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35421
2022-12-01T10:52:42.6799557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6800012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6800587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6801045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6801619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6802065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6802613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6803086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6803541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:42.6804044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:42.6804682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6805368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6805889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:42.6806358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:42.6806695Z dist init r=1, world=2
2022-12-01T10:52:42.6806955Z dist init r=0, world=2
2022-12-01T10:52:42.6807226Z skip: Need at least 4 CUDA devices (3.411s)
2022-12-01T10:52:42.6807688Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-12-01T10:52:42.6808440Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35499
2022-12-01T10:52:42.6808972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35500
2022-12-01T10:52:42.6809557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6810005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6810574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6811040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6811683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6812141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6812774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6813219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6813672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:42.6814167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:42.6815239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6815909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6816434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:42.6816903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:42.6817261Z dist init r=0, world=2
2022-12-01T10:52:42.6817494Z dist init r=1, world=2
2022-12-01T10:52:42.6817778Z skip: Need at least 4 CUDA devices (3.412s)
2022-12-01T10:52:42.6818241Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-12-01T10:52:42.6818968Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35578
2022-12-01T10:52:42.6819498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35579
2022-12-01T10:52:42.6820100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6820553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6821108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6821580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6822156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6822580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6823150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6823611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6824060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:42.6824540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:42.6825188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6825877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6826395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:42.6826847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:42.6827192Z dist init r=1, world=2
2022-12-01T10:52:42.6827441Z dist init r=0, world=2
2022-12-01T10:52:42.6827708Z skip: Need at least 4 CUDA devices (3.411s)
2022-12-01T10:52:42.6828168Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-12-01T10:52:42.6829017Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35657
2022-12-01T10:52:42.6829570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35658
2022-12-01T10:52:42.6830239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6830683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6831252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6831703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6832276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:52:42.6832714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:52:42.6833280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:52:42.6833724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:52:42.6834174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:52:42.6834674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:52:42.6835305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6835992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:52:42.6836510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:52:42.6836976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:52:42.6837315Z dist init r=1, world=2
2022-12-01T10:52:42.6837567Z dist init r=0, world=2
2022-12-01T10:52:42.6837852Z skip: Need at least 4 CUDA devices (3.411s)
2022-12-01T10:52:42.6838044Z 
2022-12-01T10:52:42.6838299Z ----------------------------------------------------------------------
2022-12-01T10:52:42.6838631Z Ran 5 tests in 18.645s
2022-12-01T10:52:42.6838787Z 
2022-12-01T10:52:42.6838894Z OK (skipped=5)
2022-12-01T10:52:42.6839043Z 
2022-12-01T10:52:42.6839165Z Generating XML reports...
2022-12-01T10:52:42.6839774Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221201105223.xml
2022-12-01T10:52:42.6840148Z 
2022-12-01T10:52:42.6840491Z ##[endgroup]
2022-12-01T10:52:42.6841120Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_1b_s9uij)
2022-12-01T10:52:42.6841494Z 
2022-12-01T10:52:42.6841743Z Running distributed/fsdp/test_fsdp_memory ... [2022-12-01 10:52:42.677765]
2022-12-01T10:52:42.6842421Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:52:42.678033]
2022-12-01T10:53:00.0928233Z 
2022-12-01T10:53:00.0929003Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory
2022-12-01T10:53:00.0929958Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_oyexrzpf)
2022-12-01T10:53:00.0930346Z 
2022-12-01T10:53:00.0930427Z Running tests...
2022-12-01T10:53:00.0930930Z ----------------------------------------------------------------------
2022-12-01T10:53:00.0931557Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2022-12-01T10:53:00.0932022Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:53:00.0932522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35771
2022-12-01T10:53:00.0933188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35772
2022-12-01T10:53:00.0933804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:00.0934680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:00.0935208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:00.0935663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:00.0936240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:00.0936688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:00.0937240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:00.0937707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:00.0938159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:53:00.0938644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:53:00.0939299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:00.0939991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:00.0940510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:53:00.0940960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:00.0942176Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:00.0942939Z   warnings.warn(
2022-12-01T10:53:00.0944044Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:00.0944775Z   warnings.warn(
2022-12-01T10:53:00.0945532Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:00.0946054Z   warnings.warn(
2022-12-01T10:53:00.0946798Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:00.0947334Z   warnings.warn(
2022-12-01T10:53:00.0948092Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:53:00.0948619Z   warnings.warn(
2022-12-01T10:53:00.0949470Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:53:00.0950037Z   warnings.warn(
2022-12-01T10:53:00.0950264Z dist init r=0, world=2
2022-12-01T10:53:00.0950512Z dist init r=1, world=2
2022-12-01T10:53:00.0950826Z ok (8.870s)
2022-12-01T10:53:00.0951278Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35884
2022-12-01T10:53:00.0951783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35885
2022-12-01T10:53:00.0952398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:00.0952845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:00.0953409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:00.0953839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:00.0954415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:00.0954884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:00.0955456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:00.0955930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:00.0956429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:53:00.0956926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:53:00.0957564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:00.0958248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:00.0958772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:53:00.0959243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:00.0960446Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:00.0961197Z   warnings.warn(
2022-12-01T10:53:00.0962303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:00.0963040Z   warnings.warn(
2022-12-01T10:53:00.0963786Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:00.0964306Z   warnings.warn(
2022-12-01T10:53:00.0965041Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:00.0965573Z   warnings.warn(
2022-12-01T10:53:00.0966395Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:53:00.0966942Z   warnings.warn(
2022-12-01T10:53:00.0967754Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:53:00.0968293Z   warnings.warn(
2022-12-01T10:53:00.0968538Z dist init r=1, world=2
2022-12-01T10:53:00.0968767Z dist init r=0, world=2
2022-12-01T10:53:00.0969001Z ok (6.718s)
2022-12-01T10:53:00.0969146Z 
2022-12-01T10:53:00.0969417Z ----------------------------------------------------------------------
2022-12-01T10:53:00.0969722Z Ran 2 tests in 15.589s
2022-12-01T10:53:00.0969878Z 
2022-12-01T10:53:00.0969970Z OK
2022-12-01T10:53:00.0970105Z 
2022-12-01T10:53:00.0970228Z Generating XML reports...
2022-12-01T10:53:00.0970800Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221201105244.xml
2022-12-01T10:53:00.0971145Z 
2022-12-01T10:53:00.0971459Z ##[endgroup]
2022-12-01T10:53:00.0972047Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_oyexrzpf)
2022-12-01T10:53:00.0972398Z 
2022-12-01T10:53:00.0972654Z Running distributed/fsdp/test_fsdp_overlap ... [2022-12-01 10:53:00.092957]
2022-12-01T10:53:00.0973340Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:53:00.093315]
2022-12-01T10:53:23.6637002Z 
2022-12-01T10:53:23.6637562Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap
2022-12-01T10:53:23.6641349Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_q80fzx_k)
2022-12-01T10:53:23.6641740Z 
2022-12-01T10:53:23.6641880Z Running tests...
2022-12-01T10:53:23.6642445Z ----------------------------------------------------------------------
2022-12-01T10:53:23.6643013Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-12-01T10:53:23.6643933Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:53:23.6645064Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36032
2022-12-01T10:53:23.6646520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:23.6646995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:23.6647571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:23.6648046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:23.6648584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:53:23.6649182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:53:23.6649744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:23.6650924Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:23.6651812Z   warnings.warn(
2022-12-01T10:53:23.6652720Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:23.6653301Z   warnings.warn(
2022-12-01T10:53:23.6654291Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:53:23.6655872Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:53:23.6657057Z [W python_variable.cpp:327] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-12-01T10:53:23.6657688Z dist init r=0, world=1
2022-12-01T10:53:23.6657834Z 
2022-12-01T10:53:23.6657928Z rank0:
2022-12-01T10:53:23.6658385Z   e1: {'cpu_iter': 0.0018716594000000697, 'cpu_wait': 3.5185700000184285e-05, 'gpu_compute': 0.053331200033426285, 'gpu_total': 0.7462944030761719}
2022-12-01T10:53:23.6658973Z   e2: {'cpu_iter': 0.004790265999999921, 'cpu_wait': 3.3901800000091244e-05, 'gpu_compute': 0.2370111994445324, 'gpu_total': 2.160636806488037}
2022-12-01T10:53:23.6659578Z   e3: {'cpu_iter': 0.002067133900000062, 'cpu_wait': 0.39387446090000006, 'gpu_compute': 396.3637062072754, 'gpu_total': 396.645849609375}
2022-12-01T10:53:23.6660136Z   e4: {'cpu_iter': 0.005028552799999986, 'cpu_wait': 0.39185288200000096, 'gpu_compute': 396.39857788085936, 'gpu_total': 396.9615844726562}
2022-12-01T10:53:23.6660456Z ok (21.727s)
2022-12-01T10:53:23.6661489Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-12-01T10:53:23.6662098Z 
2022-12-01T10:53:23.6662365Z ----------------------------------------------------------------------
2022-12-01T10:53:23.6662678Z Ran 2 tests in 21.728s
2022-12-01T10:53:23.6662838Z 
2022-12-01T10:53:23.6662946Z OK (skipped=1)
2022-12-01T10:53:23.6663096Z 
2022-12-01T10:53:23.6663218Z Generating XML reports...
2022-12-01T10:53:23.6663850Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221201105301.xml
2022-12-01T10:53:23.6664716Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221201105301.xml
2022-12-01T10:53:23.6665108Z 
2022-12-01T10:53:23.6665494Z ##[endgroup]
2022-12-01T10:53:23.6666106Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_q80fzx_k)
2022-12-01T10:53:23.6666462Z 
2022-12-01T10:53:23.6666710Z Running distributed/fsdp/test_fsdp_input ... [2022-12-01 10:53:23.663820]
2022-12-01T10:53:23.6667381Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:53:23.664110]
2022-12-01T10:53:34.5319467Z 
2022-12-01T10:53:34.5320265Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input
2022-12-01T10:53:34.5322216Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_p88wqnzh)
2022-12-01T10:53:34.5322734Z 
2022-12-01T10:53:34.5323123Z Running tests...
2022-12-01T10:53:34.5323675Z ----------------------------------------------------------------------
2022-12-01T10:53:34.5324224Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2022-12-01T10:53:34.5324779Z   test_input_type_dict (__main__.TestInput)
2022-12-01T10:53:34.5325225Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:53:34.5325708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36109
2022-12-01T10:53:34.5326386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:34.5326877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:34.5327465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:34.5327952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:34.5328433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:53:34.5329135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:53:34.5329674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:34.5330967Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:34.5331798Z   warnings.warn(
2022-12-01T10:53:34.5332593Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:34.5333187Z   warnings.warn(
2022-12-01T10:53:34.5333419Z dist init r=0, world=1
2022-12-01T10:53:34.5333672Z ok (5.293s)
2022-12-01T10:53:34.5333960Z   test_input_type_list (__main__.TestInput)
2022-12-01T10:53:34.5334776Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36151
2022-12-01T10:53:34.5335529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:34.5336032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:34.5336645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:34.5337133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:34.5337610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:53:34.5338317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T10:53:34.5338876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:34.5340151Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:34.5340966Z   warnings.warn(
2022-12-01T10:53:34.5341868Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:34.5342525Z   warnings.warn(
2022-12-01T10:53:34.5342765Z dist init r=0, world=1
2022-12-01T10:53:34.5343033Z ok (3.709s)
2022-12-01T10:53:34.5343175Z 
2022-12-01T10:53:34.5343476Z ----------------------------------------------------------------------
2022-12-01T10:53:34.5343805Z Ran 2 tests in 9.002s
2022-12-01T10:53:34.5343979Z 
2022-12-01T10:53:34.5344076Z OK
2022-12-01T10:53:34.5344215Z 
2022-12-01T10:53:34.5344343Z Generating XML reports...
2022-12-01T10:53:34.5344941Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221201105325.xml
2022-12-01T10:53:34.5345272Z 
2022-12-01T10:53:34.5345609Z ##[endgroup]
2022-12-01T10:53:34.5346244Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_p88wqnzh)
2022-12-01T10:53:34.5346637Z 
2022-12-01T10:53:34.5346946Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-12-01 10:53:34.532110]
2022-12-01T10:53:34.5347719Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:53:34.532428]
2022-12-01T10:53:43.7333404Z 
2022-12-01T10:53:43.7334367Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec
2022-12-01T10:53:43.7335722Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_hr1wri3i)
2022-12-01T10:53:43.7336313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyqtdnh5
2022-12-01T10:53:43.7336887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyqtdnh5/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7337220Z 
2022-12-01T10:53:43.7337333Z Running tests...
2022-12-01T10:53:43.7337848Z ----------------------------------------------------------------------
2022-12-01T10:53:43.7338470Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-12-01T10:53:43.7339189Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:53:43.7339752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36228
2022-12-01T10:53:43.7340253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36229
2022-12-01T10:53:43.7340664Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36230
2022-12-01T10:53:43.7341102Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36231
2022-12-01T10:53:43.7341744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7342204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7342965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7343407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7343985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7344379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7344961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7345408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7345983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7346681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7347285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7347900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7348464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7348909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7349478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7349942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7350390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaiv3chxc
2022-12-01T10:53:43.7350921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1x5x7jj1
2022-12-01T10:53:43.7351450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaiv3chxc/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7351983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1x5x7jj1/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7352513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewf8y0pv
2022-12-01T10:53:43.7353046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewf8y0pv/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7353556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:43.7354011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:53:43.7354465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:53:43.7354960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ezmmw1p
2022-12-01T10:53:43.7355477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ezmmw1p/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7355982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:53:43.7356326Z ok (3.479s)
2022-12-01T10:53:43.7356661Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-12-01T10:53:43.7357152Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36364
2022-12-01T10:53:43.7357650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36365
2022-12-01T10:53:43.7358095Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36366
2022-12-01T10:53:43.7358519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36367
2022-12-01T10:53:43.7359135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7359586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7360159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7360614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7361193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7361689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7362253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7362679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7363250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7363789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7364369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7364891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7365468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7365912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7366465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7366928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7367390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0egaewm
2022-12-01T10:53:43.7367915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0egaewm/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7368447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy05w6h8g
2022-12-01T10:53:43.7368979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy05w6h8g/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7369488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:53:43.7369965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazlzkc1t
2022-12-01T10:53:43.7370496Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazlzkc1t/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7371019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_64wtnf
2022-12-01T10:53:43.7371546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_64wtnf/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7372030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:53:43.7372502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:43.7372967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:53:43.7373343Z skip: Need at least 4 CUDA devices (1.909s)
2022-12-01T10:53:43.7373722Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-12-01T10:53:43.7374231Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36500
2022-12-01T10:53:43.7375110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36501
2022-12-01T10:53:43.7375539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 36502
2022-12-01T10:53:43.7375973Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 36503
2022-12-01T10:53:43.7376594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7377034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7377608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7378085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7378663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7379088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7379656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7380115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7380688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7381210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7381794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7382326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7382882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:43.7383326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:43.7383893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:43.7384351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:43.7384791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv4c_tr7y
2022-12-01T10:53:43.7385331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv4c_tr7y/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7385858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c1o8xt3
2022-12-01T10:53:43.7386368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c1o8xt3/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7386876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T10:53:43.7387369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjd6t080
2022-12-01T10:53:43.7387899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjd6t080/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7388384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:43.7388878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqg87c4a0
2022-12-01T10:53:43.7389404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqg87c4a0/_remote_module_non_scriptable.py
2022-12-01T10:53:43.7389905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T10:53:43.7390353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:53:43.7390747Z skip: Need at least 4 CUDA devices (1.909s)
2022-12-01T10:53:43.7391113Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.012s)
2022-12-01T10:53:43.7391491Z   test_device_placement (__main__.TestShardingSpec) ... ok (0.007s)
2022-12-01T10:53:43.7391889Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.007s)
2022-12-01T10:53:43.7392302Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-12-01T10:53:43.7392686Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-12-01T10:53:43.7393070Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-12-01T10:53:43.7393483Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.010s)
2022-12-01T10:53:43.7393741Z 
2022-12-01T10:53:43.7394019Z ----------------------------------------------------------------------
2022-12-01T10:53:43.7394332Z Ran 10 tests in 7.338s
2022-12-01T10:53:43.7394501Z 
2022-12-01T10:53:43.7394610Z OK (skipped=2)
2022-12-01T10:53:43.7394764Z 
2022-12-01T10:53:43.7394888Z Generating XML reports...
2022-12-01T10:53:43.7395517Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221201105336.xml
2022-12-01T10:53:43.7396347Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221201105336.xml
2022-12-01T10:53:43.7396711Z 
2022-12-01T10:53:43.7397041Z ##[endgroup]
2022-12-01T10:53:43.7397708Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_hr1wri3i)
2022-12-01T10:53:43.7398105Z 
2022-12-01T10:53:43.7398432Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2022-12-01 10:53:43.733722]
2022-12-01T10:53:43.7399132Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:53:43.734082]
2022-12-01T10:53:51.0725875Z 
2022-12-01T10:53:51.0726385Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_pure_fp16
2022-12-01T10:53:51.0727477Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_dgh31c48)
2022-12-01T10:53:51.0727843Z 
2022-12-01T10:53:51.0727958Z Running tests...
2022-12-01T10:53:51.0728469Z ----------------------------------------------------------------------
2022-12-01T10:53:51.0729062Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16
2022-12-01T10:53:51.0729552Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2022-12-01T10:53:51.0730075Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:53:51.0731131Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/73315 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.597s)
2022-12-01T10:53:51.0731856Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16)
2022-12-01T10:53:51.0732535Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36671
2022-12-01T10:53:51.0733053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36672
2022-12-01T10:53:51.0733671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:51.0734126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:51.0737480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:51.0737949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:51.0738532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:51.0738986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:51.0739539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:51.0740008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:51.0740464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:53:51.0740963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:53:51.0741604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:51.0742299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:51.0742819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:53:51.0743294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:51.0744742Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:51.0745521Z   warnings.warn(
2022-12-01T10:53:51.0746623Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:51.0747465Z   warnings.warn(
2022-12-01T10:53:51.0747714Z dist init r=0, world=2
2022-12-01T10:53:51.0747946Z dist init r=1, world=2
2022-12-01T10:53:51.0748181Z ok (3.912s)
2022-12-01T10:53:51.0748327Z 
2022-12-01T10:53:51.0748598Z ----------------------------------------------------------------------
2022-12-01T10:53:51.0748911Z Ran 2 tests in 5.509s
2022-12-01T10:53:51.0749070Z 
2022-12-01T10:53:51.0749180Z OK (skipped=1)
2022-12-01T10:53:51.0749335Z 
2022-12-01T10:53:51.0749461Z Generating XML reports...
2022-12-01T10:53:51.0750026Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221201105345.xml
2022-12-01T10:53:51.0750375Z 
2022-12-01T10:53:51.0750694Z ##[endgroup]
2022-12-01T10:53:51.0751301Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_dgh31c48)
2022-12-01T10:53:51.0751657Z 
2022-12-01T10:53:51.0751953Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-12-01 10:53:51.072527]
2022-12-01T10:53:51.0752656Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:53:51.072799]
2022-12-01T10:53:58.4314960Z 
2022-12-01T10:53:58.4315673Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2022-12-01T10:53:58.4317159Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_afd7lntz)
2022-12-01T10:53:58.4317700Z 
2022-12-01T10:53:58.4317810Z Running tests...
2022-12-01T10:53:58.4318596Z ----------------------------------------------------------------------
2022-12-01T10:53:58.4319192Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-12-01T10:53:58.4319652Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-12-01T10:53:58.4320118Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:53:58.4320615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36789
2022-12-01T10:53:58.4321064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36790
2022-12-01T10:53:58.4321628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:58.4322141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:58.4322688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:58.4323228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:58.4324233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:53:58.4325002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:53:58.4325541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:53:58.4326008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:53:58.4326685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:53:58.4327179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:53:58.4327865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:58.4328714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:53:58.4329265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:53:58.4329746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:53:58.4331037Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:58.4331854Z   warnings.warn(
2022-12-01T10:53:58.4333024Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:53:58.4333823Z   warnings.warn(
2022-12-01T10:53:58.4334969Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:58.4335538Z   warnings.warn(
2022-12-01T10:53:58.4336332Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:53:58.4336907Z   warnings.warn(
2022-12-01T10:53:58.4337715Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:53:58.4338298Z   warnings.warn(
2022-12-01T10:53:58.4339093Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:53:58.4339677Z   warnings.warn(
2022-12-01T10:53:58.4339931Z dist init r=1, world=2
2022-12-01T10:53:58.4340173Z dist init r=0, world=2
2022-12-01T10:53:58.4340422Z ok (5.500s)
2022-12-01T10:53:58.4340577Z 
2022-12-01T10:53:58.4340864Z ----------------------------------------------------------------------
2022-12-01T10:53:58.4341195Z Ran 1 test in 5.500s
2022-12-01T10:53:58.4341362Z 
2022-12-01T10:53:58.4341455Z OK
2022-12-01T10:53:58.4341593Z 
2022-12-01T10:53:58.4341718Z Generating XML reports...
2022-12-01T10:53:58.4342377Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221201105352.xml
2022-12-01T10:53:58.4342759Z 
2022-12-01T10:53:58.4343086Z ##[endgroup]
2022-12-01T10:53:58.4343760Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_afd7lntz)
2022-12-01T10:53:58.4344175Z 
2022-12-01T10:53:58.4344463Z Running distributed/fsdp/test_fsdp_traversal ... [2022-12-01 10:53:58.431496]
2022-12-01T10:53:58.4345282Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:53:58.431752]
2022-12-01T10:54:05.1907272Z 
2022-12-01T10:54:05.1925844Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal
2022-12-01T10:54:05.1926899Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_ooqsq_xm)
2022-12-01T10:54:05.1927315Z 
2022-12-01T10:54:05.1927435Z Running tests...
2022-12-01T10:54:05.1927950Z ----------------------------------------------------------------------
2022-12-01T10:54:05.1928547Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-12-01T10:54:05.1929064Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:54:05.1929528Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36907
2022-12-01T10:54:05.1930025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36908
2022-12-01T10:54:05.1930694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:05.1931179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:05.1931771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:05.1932274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:05.1932888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:05.1933348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:05.1933951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:05.1934858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:05.1935719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:05.1936427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:05.1937129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:05.1937829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:05.1938357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:05.1938811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:05.1939161Z dist init r=0, world=2
2022-12-01T10:54:05.1939413Z dist init r=1, world=2
2022-12-01T10:54:05.1939635Z ok (4.898s)
2022-12-01T10:54:05.1939783Z 
2022-12-01T10:54:05.1940058Z ----------------------------------------------------------------------
2022-12-01T10:54:05.1940389Z Ran 1 test in 4.898s
2022-12-01T10:54:05.1940548Z 
2022-12-01T10:54:05.1940626Z OK
2022-12-01T10:54:05.1940759Z 
2022-12-01T10:54:05.1940883Z Generating XML reports...
2022-12-01T10:54:05.1941474Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221201105359.xml
2022-12-01T10:54:05.1941818Z 
2022-12-01T10:54:05.1942123Z ##[endgroup]
2022-12-01T10:54:05.1942736Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_ooqsq_xm)
2022-12-01T10:54:05.1943105Z 
2022-12-01T10:54:05.1943360Z Running distributed/fsdp/test_utils ... [2022-12-01 10:54:05.190756]
2022-12-01T10:54:05.1944019Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:54:05.191005]
2022-12-01T10:54:08.5191008Z 
2022-12-01T10:54:08.5191737Z Expand the folded group to see the log file of distributed/fsdp/test_utils
2022-12-01T10:54:08.5192799Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_2zaxysqu)
2022-12-01T10:54:08.5193316Z 
2022-12-01T10:54:08.5193501Z Running tests...
2022-12-01T10:54:08.5194208Z ----------------------------------------------------------------------
2022-12-01T10:54:08.5194781Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2022-12-01T10:54:08.5195264Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (1.538s)
2022-12-01T10:54:08.5196003Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2022-12-01T10:54:08.5196461Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.003s)
2022-12-01T10:54:08.5196814Z   test_packed_sequence (__main__.TestUtils)
2022-12-01T10:54:08.5197196Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.003s)
2022-12-01T10:54:08.5197573Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2022-12-01T10:54:08.5197793Z 
2022-12-01T10:54:08.5198062Z ----------------------------------------------------------------------
2022-12-01T10:54:08.5198391Z Ran 5 tests in 1.549s
2022-12-01T10:54:08.5198553Z 
2022-12-01T10:54:08.5198647Z OK
2022-12-01T10:54:08.5198762Z 
2022-12-01T10:54:08.5198887Z Generating XML reports...
2022-12-01T10:54:08.5199449Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221201105406.xml
2022-12-01T10:54:08.5199771Z 
2022-12-01T10:54:08.5200087Z ##[endgroup]
2022-12-01T10:54:08.5200636Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_2zaxysqu)
2022-12-01T10:54:08.5200977Z 
2022-12-01T10:54:08.5201323Z Running distributed/_shard/checkpoint/test_file_system_checkpoint_cpu ... [2022-12-01 10:54:08.519063]
2022-12-01T10:54:08.5202099Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:54:08.519324]
2022-12-01T10:54:21.6184606Z 
2022-12-01T10:54:21.6185547Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_file_system_checkpoint_cpu
2022-12-01T10:54:21.6186902Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_cpu_ss_y7c8l)
2022-12-01T10:54:21.6187617Z 
2022-12-01T10:54:21.6187742Z Running tests...
2022-12-01T10:54:21.6188222Z ----------------------------------------------------------------------
2022-12-01T10:54:21.6189164Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu
2022-12-01T10:54:21.6189873Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:54:21.6190452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37055
2022-12-01T10:54:21.6190947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37056
2022-12-01T10:54:21.6191821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6192228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6192826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6193325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6193923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6194418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6195247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6195819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6196258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:21.6196751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:21.6197233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:21.6197729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:21.6198341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6199038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6199416Z ok (3.506s)
2022-12-01T10:54:21.6199893Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37129
2022-12-01T10:54:21.6200464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37130
2022-12-01T10:54:21.6201074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6201509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6202092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6202564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6203147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6203580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6204157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6204633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6205057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:21.6205531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:21.6206025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:21.6206521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:21.6207166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6207858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6208256Z ok (2.009s)
2022-12-01T10:54:21.6208694Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37203
2022-12-01T10:54:21.6209237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37204
2022-12-01T10:54:21.6209847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6210298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6210855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6211322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6211973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6212433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6212992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6213519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6213958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:21.6214686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:21.6215198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:21.6215682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:21.6216356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6217029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6217428Z ok (1.807s)
2022-12-01T10:54:21.6217915Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37277
2022-12-01T10:54:21.6218486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37278
2022-12-01T10:54:21.6219088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6219542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6220119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6220571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6221157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6221605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6222182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6222629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6223071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:21.6223548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:21.6224021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:21.6224515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:21.6225175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6225862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6226241Z ok (2.108s)
2022-12-01T10:54:21.6226610Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.018s)
2022-12-01T10:54:21.6227261Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37351
2022-12-01T10:54:21.6227879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37352
2022-12-01T10:54:21.6228474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6228933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6229607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6230095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6230742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:21.6231193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:21.6231765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:21.6232212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:21.6232656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:21.6233133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:21.6233628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:21.6234109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:21.6234769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6235457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:21.6235865Z ok (1.807s)
2022-12-01T10:54:21.6235997Z 
2022-12-01T10:54:21.6236269Z ----------------------------------------------------------------------
2022-12-01T10:54:21.6236601Z Ran 6 tests in 11.256s
2022-12-01T10:54:21.6236765Z 
2022-12-01T10:54:21.6236860Z OK
2022-12-01T10:54:21.6236992Z 
2022-12-01T10:54:21.6237099Z Generating XML reports...
2022-12-01T10:54:21.6237798Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221201105410.xml
2022-12-01T10:54:21.6238757Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221201105410.xml
2022-12-01T10:54:21.6239807Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201105410.xml
2022-12-01T10:54:21.6240304Z 
2022-12-01T10:54:21.6240622Z ##[endgroup]
2022-12-01T10:54:21.6241338Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_cpu_ss_y7c8l)
2022-12-01T10:54:21.6241769Z 
2022-12-01T10:54:21.6242066Z Running distributed/fsdp/test_fsdp_param_exec_order_wrap ... [2022-12-01 10:54:21.618473]
2022-12-01T10:54:21.6242799Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_param_exec_order_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:54:21.618744]
2022-12-01T10:54:39.9972244Z 
2022-12-01T10:54:39.9973132Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_param_exec_order_wrap
2022-12-01T10:54:39.9975188Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_param_exec_order_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_param_exec_order_wrap_1oxqpmbs)
2022-12-01T10:54:39.9976071Z 
2022-12-01T10:54:39.9976285Z Running tests...
2022-12-01T10:54:39.9977101Z ----------------------------------------------------------------------
2022-12-01T10:54:39.9977710Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap
2022-12-01T10:54:39.9978276Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-12-01T10:54:39.9978835Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:54:39.9979577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37460
2022-12-01T10:54:39.9980059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37461
2022-12-01T10:54:39.9980793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:39.9981232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:39.9981819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:39.9982296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:39.9982883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:39.9983319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:39.9983906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:39.9984380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:39.9984846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:39.9985330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:39.9985993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:39.9986691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:39.9987202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:39.9987678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:39.9988905Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:39.9989671Z   warnings.warn(
2022-12-01T10:54:39.9990791Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:39.9991532Z   warnings.warn(
2022-12-01T10:54:39.9992284Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:54:39.9992834Z   warnings.warn(
2022-12-01T10:54:39.9993582Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:54:39.9994124Z   warnings.warn(
2022-12-01T10:54:39.9994871Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:54:39.9995424Z   warnings.warn(
2022-12-01T10:54:39.9996259Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:54:39.9996814Z   warnings.warn(
2022-12-01T10:54:39.9997098Z dist init r=0, world=2
2022-12-01T10:54:39.9997351Z dist init r=1, world=2
2022-12-01T10:54:39.9997596Z ok (5.611s)
2022-12-01T10:54:39.9997985Z   test_fsdp_flatten_params_exec_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-12-01T10:54:39.9998600Z Test ``_fsdp_params_exec_order`` with ``ParamExecOrderWrapPolicy``, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37543
2022-12-01T10:54:39.9999148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37544
2022-12-01T10:54:39.9999751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:40.0000205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:40.0000789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:40.0001262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:40.0001831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:40.0002283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:40.0002858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:40.0003309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:40.0003769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:40.0004271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:40.0004938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:40.0005617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:40.0006152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:40.0006628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:40.0007858Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:40.0008616Z   warnings.warn(
2022-12-01T10:54:40.0009715Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:40.0010469Z   warnings.warn(
2022-12-01T10:54:40.0011234Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:54:40.0011778Z   warnings.warn(
2022-12-01T10:54:40.0012572Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:54:40.0013130Z   warnings.warn(
2022-12-01T10:54:40.0013900Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:54:40.0014699Z   warnings.warn(
2022-12-01T10:54:40.0015447Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T10:54:40.0015989Z   warnings.warn(
2022-12-01T10:54:40.0016242Z dist init r=0, world=2
2022-12-01T10:54:40.0016493Z dist init r=1, world=2
2022-12-01T10:54:40.0016715Z ok (3.912s)
2022-12-01T10:54:40.0017140Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-12-01T10:54:40.0017764Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37626
2022-12-01T10:54:40.0018291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37627
2022-12-01T10:54:40.0018905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:40.0019358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:40.0019936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:40.0020392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:40.0020972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:40.0021419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:40.0021986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:40.0022467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:40.0022927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:40.0023430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:40.0024078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:40.0024776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:40.0025304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:40.0025787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:40.0026995Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:40.0027751Z   warnings.warn(
2022-12-01T10:54:40.0028868Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:40.0029707Z   warnings.warn(
2022-12-01T10:54:40.0029977Z dist init r=1, world=2
2022-12-01T10:54:40.0030210Z dist init r=0, world=2
2022-12-01T10:54:40.0030451Z ok (3.510s)
2022-12-01T10:54:40.0031001Z   test_fsdp_flatten_params_exec_order_symbolic_trace_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-12-01T10:54:40.0031606Z Tests ``ParamExecOrderWrapPolicy`` with symbolic tracing. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37705
2022-12-01T10:54:40.0032156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37706
2022-12-01T10:54:40.0032777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:40.0033237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:40.0033799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:40.0034274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:40.0034857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:54:40.0035289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:54:40.0035867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:54:40.0036331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:54:40.0036788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:54:40.0037436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:40.0037976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:54:40.0038630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:54:40.0039159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:54:40.0039616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:54:40.0040832Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:40.0041583Z   warnings.warn(
2022-12-01T10:54:40.0042697Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1427: UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.
2022-12-01T10:54:40.0043441Z   warnings.warn(
2022-12-01T10:54:40.0043672Z dist init r=0, world=2
2022-12-01T10:54:40.0043924Z dist init r=1, world=2
2022-12-01T10:54:40.0044164Z ok (3.510s)
2022-12-01T10:54:40.0044294Z 
2022-12-01T10:54:40.0044566Z ----------------------------------------------------------------------
2022-12-01T10:54:40.0044896Z Ran 4 tests in 16.544s
2022-12-01T10:54:40.0045058Z 
2022-12-01T10:54:40.0045151Z OK
2022-12-01T10:54:40.0045285Z 
2022-12-01T10:54:40.0045390Z Generating XML reports...
2022-12-01T10:54:40.0046022Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20221201105423.xml
2022-12-01T10:54:40.0046393Z 
2022-12-01T10:54:40.0046780Z ##[endgroup]
2022-12-01T10:54:40.0047434Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_param_exec_order_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_param_exec_order_wrap_1oxqpmbs)
2022-12-01T10:54:40.0047882Z 
2022-12-01T10:54:40.0048191Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-12-01 10:54:39.997343]
2022-12-01T10:54:40.0048942Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:54:39.997656]
2022-12-01T11:13:32.4756817Z 
2022-12-01T11:13:32.4759272Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent
2022-12-01T11:13:32.4760713Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_jxydwv5c)
2022-12-01T11:13:32.4762281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfin1p51
2022-12-01T11:13:32.4763086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfin1p51/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4766546Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2022-12-01T11:13:32.4767099Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2022-12-01T11:13:32.4768541Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2022-12-01T11:13:32.4769801Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2022-12-01T11:13:32.4770212Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2022-12-01T11:13:32.4770764Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2022-12-01T11:13:32.4771680Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2022-12-01T11:13:32.4772519Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T11:13:32.4772978Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T11:13:32.4773798Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T11:13:32.4774387Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-12-01T11:13:32.4776204Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2022-12-01T11:13:32.4777002Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2022-12-01T11:13:32.4779437Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2022-12-01T11:13:32.4780748Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4781189Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4781590Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4782323Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4782761Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4783308Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4783708Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4784092Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-12-01T11:13:32.4799532Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2022-12-01T11:13:32.4813612Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4814115Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4815259Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4815992Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4816572Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4817145Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4817721Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4818300Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4818808Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4819363Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4819867Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4820325Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4820819Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4821353Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4821865Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4822384Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4822957Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4823398Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4824006Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4824616Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4825063Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4825549Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4826094Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4826603Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4827115Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4827628Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4828232Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4828644Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4829787Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4830254Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4830814Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4831383Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4832056Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4832485Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4833156Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4833652Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4834123Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4834544Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4835053Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4835552Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4836007Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4836553Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4837252Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4837771Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4838257Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4838844Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4839618Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4840053Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4840584Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4841117Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4841596Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4842125Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4842634Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4843087Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4843668Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4844125Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4844672Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4845200Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4845733Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4846345Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4846850Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4847341Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4847749Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4848301Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4848798Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4849214Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4849753Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4850340Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4850992Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4851679Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4852179Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4852642Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4853191Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4853644Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4854204Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4855498Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4855921Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4856463Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4856915Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4857480Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4857932Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4858458Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4859243Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4859802Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4860319Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4860773Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4861399Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-12-01T11:13:32.4862377Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2022-12-01T11:13:32.4863295Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-12-01T11:13:32.4863874Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-12-01T11:13:32.4864392Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-12-01T11:13:32.4865153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4865629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4866217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4866623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4867146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmk4_yav
2022-12-01T11:13:32.4867659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmk4_yav/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4867950Z 
2022-12-01T11:13:32.4868107Z Running tests...
2022-12-01T11:13:32.4868454Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4869097Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.4869775Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.4870336Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37852
2022-12-01T11:13:32.4870776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37853
2022-12-01T11:13:32.4871252Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 37854
2022-12-01T11:13:32.4871739Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 37855
2022-12-01T11:13:32.4872340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4872764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4873334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4873810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4874386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4874853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4875465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4875945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4876503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4876961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4877482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4878090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4878728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4879176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4879759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4880209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4880675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tdxwead
2022-12-01T11:13:32.4881227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tdxwead/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4881776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3cqfg97i
2022-12-01T11:13:32.4882291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3cqfg97i/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4882835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpc5z7xj
2022-12-01T11:13:32.4883379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpc5z7xj/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4883894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0j249xd
2022-12-01T11:13:32.4884427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0j249xd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4884914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.4885369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.4885791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.4886344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.4886681Z skip: Need at least 4 CUDA devices (3.890s)
2022-12-01T11:13:32.4886865Z 
2022-12-01T11:13:32.4887220Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4887547Z Ran 1 test in 3.891s
2022-12-01T11:13:32.4887761Z 
2022-12-01T11:13:32.4887874Z OK (skipped=1)
2022-12-01T11:13:32.4888025Z 
2022-12-01T11:13:32.4888206Z Generating XML reports...
2022-12-01T11:13:32.4888877Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221201105443.xml
2022-12-01T11:13:32.4889600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4890112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4890699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4891114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4891575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_2725ju
2022-12-01T11:13:32.4892172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_2725ju/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4892404Z 
2022-12-01T11:13:32.4892577Z Running tests...
2022-12-01T11:13:32.4892967Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4893528Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.4894047Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.4895339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38023
2022-12-01T11:13:32.4895868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38024
2022-12-01T11:13:32.4896299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38025
2022-12-01T11:13:32.4896772Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38026
2022-12-01T11:13:32.4897372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4897831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4898422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4898870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4899451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4899894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4900543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4900934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4901577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4902142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4902711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4903170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4903811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4904249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4904816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4905285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4905752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsmv8a1g
2022-12-01T11:13:32.4906420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ezpwi8x
2022-12-01T11:13:32.4906930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsmv8a1g/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4907606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ezpwi8x/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4908076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfyuaf4t
2022-12-01T11:13:32.4908580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfyuaf4t/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4909091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyc6iy450
2022-12-01T11:13:32.4909627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyc6iy450/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4910136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.4910598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.4911062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.4911533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.4911931Z fi_getinfo: -61
2022-12-01T11:13:32.4912189Z fi_getinfo: -61
2022-12-01T11:13:32.4912459Z fi_getinfo: -61
2022-12-01T11:13:32.4912730Z fi_getinfo: -61
2022-12-01T11:13:32.4912944Z ok (5.998s)
2022-12-01T11:13:32.4913092Z 
2022-12-01T11:13:32.4913364Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4913692Z Ran 1 test in 5.998s
2022-12-01T11:13:32.4913834Z 
2022-12-01T11:13:32.4913927Z OK
2022-12-01T11:13:32.4914063Z 
2022-12-01T11:13:32.4914185Z Generating XML reports...
2022-12-01T11:13:32.4914867Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201105449.xml
2022-12-01T11:13:32.4915626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4916085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4916666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4917138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4917590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhs63x00
2022-12-01T11:13:32.4918134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhs63x00/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4918435Z 
2022-12-01T11:13:32.4918545Z Running tests...
2022-12-01T11:13:32.4918949Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4919518Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.4920085Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.4920603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38446
2022-12-01T11:13:32.4921035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38447
2022-12-01T11:13:32.4921480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38448
2022-12-01T11:13:32.4921926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38449
2022-12-01T11:13:32.4922545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4922980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4923554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4924084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4924656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4925157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4925726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4926177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4926730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4927202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4927803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4928260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4928820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4929277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4929854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4930303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4930774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mmul_sn
2022-12-01T11:13:32.4931303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkeuwonfq
2022-12-01T11:13:32.4931840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mmul_sn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4932372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkeuwonfq/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4932908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0dbou2b
2022-12-01T11:13:32.4933443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0dbou2b/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4933956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmllhnf4v
2022-12-01T11:13:32.4934996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmllhnf4v/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4935492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.4935970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.4936422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.4936882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.4937287Z fi_getinfo: -61
2022-12-01T11:13:32.4937549Z fi_getinfo: -61
2022-12-01T11:13:32.4937818Z fi_getinfo: -61
2022-12-01T11:13:32.4938088Z fi_getinfo: -61
2022-12-01T11:13:32.4938306Z ok (6.067s)
2022-12-01T11:13:32.4938453Z 
2022-12-01T11:13:32.4938720Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4939046Z Ran 1 test in 6.067s
2022-12-01T11:13:32.4939208Z 
2022-12-01T11:13:32.4939300Z OK
2022-12-01T11:13:32.4939413Z 
2022-12-01T11:13:32.4939538Z Generating XML reports...
2022-12-01T11:13:32.4940211Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201105458.xml
2022-12-01T11:13:32.4940985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4941421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4942082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4942567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4943099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekeysczc
2022-12-01T11:13:32.4943625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekeysczc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4943933Z 
2022-12-01T11:13:32.4944043Z Running tests...
2022-12-01T11:13:32.4944453Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4945014Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.4945596Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.4946117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38869
2022-12-01T11:13:32.4946570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38870
2022-12-01T11:13:32.4946998Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 38871
2022-12-01T11:13:32.4947437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 38872
2022-12-01T11:13:32.4948048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4948503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4949059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4949532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4950117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4950548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4951126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4951596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4952177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4952605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4953175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4953645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4954202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4954651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4955225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4955686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4956141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp4glzp0
2022-12-01T11:13:32.4956682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp4glzp0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4957215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjbrt4nz
2022-12-01T11:13:32.4957755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjbrt4nz/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4958268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gvft6ss
2022-12-01T11:13:32.4958797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gvft6ss/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4959382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_tizrk8
2022-12-01T11:13:32.4959908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_tizrk8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4960497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.4960973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.4961442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.4961893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.4962280Z fi_getinfo: -61
2022-12-01T11:13:32.4962557Z fi_getinfo: -61
2022-12-01T11:13:32.4962809Z fi_getinfo: -61
2022-12-01T11:13:32.4963076Z fi_getinfo: -61
2022-12-01T11:13:32.4963311Z ok (6.202s)
2022-12-01T11:13:32.4963457Z 
2022-12-01T11:13:32.4963709Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4964041Z Ran 1 test in 6.203s
2022-12-01T11:13:32.4964203Z 
2022-12-01T11:13:32.4964293Z OK
2022-12-01T11:13:32.4964427Z 
2022-12-01T11:13:32.4964552Z Generating XML reports...
2022-12-01T11:13:32.4965217Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201105506.xml
2022-12-01T11:13:32.4965986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4966441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4967001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4967476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4967947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvs9vu0sm
2022-12-01T11:13:32.4968495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvs9vu0sm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4968781Z 
2022-12-01T11:13:32.4968891Z Running tests...
2022-12-01T11:13:32.4969297Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4969876Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.4970429Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.4970944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39292
2022-12-01T11:13:32.4971397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39293
2022-12-01T11:13:32.4972007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4972443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4973027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4973502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4974091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4975026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4975614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4976084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4976535Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbl0cox3
2022-12-01T11:13:32.4977080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbl0cox3/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4977716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kx6g959
2022-12-01T11:13:32.4978266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kx6g959/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4978822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.4979304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.4979697Z fi_getinfo: -61
2022-12-01T11:13:32.4979959Z fi_getinfo: -61
2022-12-01T11:13:32.4980195Z ok (5.481s)
2022-12-01T11:13:32.4980342Z 
2022-12-01T11:13:32.4980612Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4980923Z Ran 1 test in 5.482s
2022-12-01T11:13:32.4981084Z 
2022-12-01T11:13:32.4981176Z OK
2022-12-01T11:13:32.4981309Z 
2022-12-01T11:13:32.4981435Z Generating XML reports...
2022-12-01T11:13:32.4982115Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105514.xml
2022-12-01T11:13:32.4982870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4983328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4983905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4984378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4984825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqc_xvrk
2022-12-01T11:13:32.4985366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqc_xvrk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4985664Z 
2022-12-01T11:13:32.4985774Z Running tests...
2022-12-01T11:13:32.4986159Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4986741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.4987316Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.4987850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39482
2022-12-01T11:13:32.4988286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39483
2022-12-01T11:13:32.4988890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4989340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4989897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4990369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4990951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.4991397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.4991955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.4992427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.4992898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnr3gp3x
2022-12-01T11:13:32.4993443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnr3gp3x/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4993959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7fj7php
2022-12-01T11:13:32.4994498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7fj7php/_remote_module_non_scriptable.py
2022-12-01T11:13:32.4995013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.4995526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.4995923Z fi_getinfo: -61
2022-12-01T11:13:32.4996269Z fi_getinfo: -61
2022-12-01T11:13:32.4996760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7fj7php/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-12-01T11:13:32.4997495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnr3gp3x/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-12-01T11:13:32.4998240Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpdnr3gp3x/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-12-01T11:13:32.4998717Z ok (5.641s)
2022-12-01T11:13:32.4998866Z 
2022-12-01T11:13:32.4999136Z ----------------------------------------------------------------------
2022-12-01T11:13:32.4999451Z Ran 1 test in 5.641s
2022-12-01T11:13:32.4999611Z 
2022-12-01T11:13:32.4999702Z OK
2022-12-01T11:13:32.4999836Z 
2022-12-01T11:13:32.4999962Z Generating XML reports...
2022-12-01T11:13:32.5000627Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105522.xml
2022-12-01T11:13:32.5001395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5001848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5002420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5002875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5003342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7egwtw7
2022-12-01T11:13:32.5003882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7egwtw7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5004184Z 
2022-12-01T11:13:32.5004278Z Running tests...
2022-12-01T11:13:32.5004689Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5005272Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5005829Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5006365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39688
2022-12-01T11:13:32.5006820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39689
2022-12-01T11:13:32.5007430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5007864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5008444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5008914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5009501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5009932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5010503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5010971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5011420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_3ayekt
2022-12-01T11:13:32.5011952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_3ayekt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5012534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgb45_g2
2022-12-01T11:13:32.5013078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgb45_g2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5013618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5014088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5014957Z fi_getinfo: -61
2022-12-01T11:13:32.5015220Z fi_getinfo: -61
2022-12-01T11:13:32.5015488Z On WorkerInfo(id=1, name=worker1):
2022-12-01T11:13:32.5033456Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f454ee9283b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f454ee8e20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) + 0x42e (0x7f45588f3f1e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x1ab8b (0x7f45588d4b8b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0x10c39a8 (0x7f45501989a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #5: <unknown function> + 0x28e2ed5 (0x7f45519b7ed5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #6: <unknown function> + 0x28e307b (0x7f45519b807b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f455a7aec33 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x1fbe7e5 (0x7f455aac97e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f455a7ebcf8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x11da60f (0x7f4559ce560f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1326 (0x7f455a066476 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2195d53 (0x7f455aca0d53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x1fbe4a8 (0x7f455aac94a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x331afa1 (0x7f455be25fa1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x331b54b (0x7f455be2654b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f455a5428b1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f455a05e13e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x23653e9 (0x7f455ae703e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f455a6aba55 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x35e10f (0x7f4566d4b10f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x35e5bb (0x7f4566d4b5bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x556f49585c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x556f494ac9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x556f494ac425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556f4958b774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xa8449f (0x7f456747149f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f456746f6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f45674729c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f4567473093 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f455d41ac14 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f45674727b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4908e23 (0x7f455d413e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f455d4149b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f455d40eaf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x49387d2 (0x7f455d4437d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f454ee80c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f457eb5bbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f459f1af6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f459eed861f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-12-01T11:13:32.5043946Z Traceback (most recent call last):
2022-12-01T11:13:32.5044466Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-12-01T11:13:32.5044929Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-12-01T11:13:32.5045515Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 91, in _create_module
2022-12-01T11:13:32.5045903Z     module.to(device)
2022-12-01T11:13:32.5046344Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 987, in to
2022-12-01T11:13:32.5046718Z     return self._apply(convert)
2022-12-01T11:13:32.5047195Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 662, in _apply
2022-12-01T11:13:32.5047550Z     param_applied = fn(param)
2022-12-01T11:13:32.5048030Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 985, in convert
2022-12-01T11:13:32.5048496Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-12-01T11:13:32.5048904Z RuntimeError: CUDA error: invalid device ordinal
2022-12-01T11:13:32.5049337Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-12-01T11:13:32.5049793Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-12-01T11:13:32.5050286Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-12-01T11:13:32.5051140Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f454ee9283b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5052125Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f454ee8e20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5053333Z frame #2: c10::cuda::c10_cuda_check_implementation(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) + 0x42e (0x7f45588f3f1e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-12-01T11:13:32.5054186Z frame #3: <unknown function> + 0x1ab8b (0x7f45588d4b8b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-12-01T11:13:32.5055403Z frame #4: <unknown function> + 0x10c39a8 (0x7f45501989a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5056059Z frame #5: <unknown function> + 0x28e2ed5 (0x7f45519b7ed5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5056690Z frame #6: <unknown function> + 0x28e307b (0x7f45519b807b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5057719Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f455a7aec33 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5058565Z frame #8: <unknown function> + 0x1fbe7e5 (0x7f455aac97e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5059532Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f455a7ebcf8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5060340Z frame #10: <unknown function> + 0x11da60f (0x7f4559ce560f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5061276Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1326 (0x7f455a066476 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5062081Z frame #12: <unknown function> + 0x2195d53 (0x7f455aca0d53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5063100Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5063952Z frame #14: <unknown function> + 0x1fbe4a8 (0x7f455aac94a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5064959Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5065850Z frame #16: <unknown function> + 0x331afa1 (0x7f455be25fa1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5066488Z frame #17: <unknown function> + 0x331b54b (0x7f455be2654b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5067438Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f455a5428b1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5068547Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f455a05e13e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5069434Z frame #20: <unknown function> + 0x23653e9 (0x7f455ae703e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5070411Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f455a6aba55 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5071328Z frame #22: <unknown function> + 0x35e10f (0x7f4566d4b10f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5071977Z frame #23: <unknown function> + 0x35e5bb (0x7f4566d4b5bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5072442Z frame #24: <unknown function> + 0x1ddc68 (0x556f49585c68 in /opt/conda/bin/python)
2022-12-01T11:13:32.5072830Z frame #25: <unknown function> + 0x1049f3 (0x556f494ac9f3 in /opt/conda/bin/python)
2022-12-01T11:13:32.5073228Z frame #26: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5073624Z frame #27: <unknown function> + 0x104425 (0x556f494ac425 in /opt/conda/bin/python)
2022-12-01T11:13:32.5074017Z frame #28: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5074388Z frame #29: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)
2022-12-01T11:13:32.5074779Z frame #30: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)
2022-12-01T11:13:32.5075171Z frame #31: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5075553Z frame #32: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)
2022-12-01T11:13:32.5075923Z frame #33: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)
2022-12-01T11:13:32.5076311Z frame #34: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5076705Z frame #35: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)
2022-12-01T11:13:32.5077105Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556f4958b774 in /opt/conda/bin/python)
2022-12-01T11:13:32.5077509Z frame #37: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5077895Z frame #38: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)
2022-12-01T11:13:32.5078492Z frame #39: <unknown function> + 0xa8449f (0x7f456747149f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5079270Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f456746f6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5080277Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f45674729c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5081405Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f4567473093 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5082632Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f455d41ac14 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5083896Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f45674727b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5084835Z frame #45: <unknown function> + 0x4908e23 (0x7f455d413e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5085758Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f455d4149b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5086905Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f455d40eaf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5087695Z frame #48: <unknown function> + 0x49387d2 (0x7f455d4437d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5088381Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f454ee80c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5088874Z frame #50: <unknown function> + 0xdbbf4 (0x7f457eb5bbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-12-01T11:13:32.5089426Z frame #51: <unknown function> + 0x76db (0x7f459f1af6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-12-01T11:13:32.5089931Z frame #52: clone + 0x3f (0x7f459eed861f in /lib/x86_64-linux-gnu/libc.so.6)
2022-12-01T11:13:32.5090157Z 
2022-12-01T11:13:32.5090176Z 
2022-12-01T11:13:32.5090310Z On WorkerInfo(id=1, name=worker1):
2022-12-01T11:13:32.5128665Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f454ee9283b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f454ee8e20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) + 0x42e (0x7f45588f3f1e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x1ab8b (0x7f45588d4b8b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0x10c39a8 (0x7f45501989a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #5: <unknown function> + 0x28e2ed5 (0x7f45519b7ed5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #6: <unknown function> + 0x28e307b (0x7f45519b807b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f455a7aec33 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x1fbe7e5 (0x7f455aac97e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f455a7ebcf8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x11da60f (0x7f4559ce560f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1326 (0x7f455a066476 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2195d53 (0x7f455aca0d53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x1fbe4a8 (0x7f455aac94a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x331afa1 (0x7f455be25fa1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x331b54b (0x7f455be2654b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f455a5428b1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f455a05e13e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x23653e9 (0x7f455ae703e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f455a6aba55 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x35e10f (0x7f4566d4b10f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x35e5bb (0x7f4566d4b5bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x556f49585c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x556f494ac9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x556f494ac425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556f4958b774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xa8449f (0x7f456747149f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f456746f6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f45674729c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f4567473093 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f455d41ac14 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f45674727b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4908e23 (0x7f455d413e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f455d4149b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f455d40eaf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x49387d2 (0x7f455d4437d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f454ee80c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f457eb5bbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f459f1af6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f459eed861f in /lib/x86_64-linux-gnu/libc.so.6)\n\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 91, in _create_module\n    module.to(device)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 987, in to\n    return self._apply(convert)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 662, in _apply\n    param_applied = fn(param)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 985, in convert\n    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f454ee9283b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f454ee8e20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) + 0x42e (0x7f45588f3f1e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x1ab8b (0x7f45588d4b8b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0x10c39a8 (0x7f45501989a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #5: <unknown function> + 0x28e2ed5 (0x7f45519b7ed5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #6: <unknown function> + 0x28e307b (0x7f45519b807b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f455a7aec33 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x1fbe7e5 (0x7f455aac97e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f455a7ebcf8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x11da60f (0x7f4559ce560f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1326 (0x7f455a066476 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2195d53 (0x7f455aca0d53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x1fbe4a8 (0x7f455aac94a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x331afa1 (0x7f455be25fa1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x331b54b (0x7f455be2654b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f455a5428b1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f455a05e13e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x23653e9 (0x7f455ae703e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f455a6aba55 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x35e10f (0x7f4566d4b10f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x35e5bb (0x7f4566d4b5bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x556f49585c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x556f494ac9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x556f494ac425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556f4958b774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xa8449f (0x7f456747149f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f456746f6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f45674729c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f4567473093 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f455d41ac14 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f45674727b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4908e23 (0x7f455d413e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f455d4149b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f455d40eaf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x49387d2 (0x7f455d4437d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f454ee80c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f457eb5bbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f459f1af6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f459eed861f in /lib/x86_64-linux-gnu/libc.so.6)\n\n')
2022-12-01T11:13:32.5151526Z Traceback (most recent call last):
2022-12-01T11:13:32.5152073Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-12-01T11:13:32.5152535Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-12-01T11:13:32.5152964Z   File "/tmp/tmpe7egwtw7/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-12-01T11:13:32.5153317Z     module = module_rref.local_value()
2022-12-01T11:13:32.5153857Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 220, in _handle_exception
2022-12-01T11:13:32.5154438Z     raise result.exception_type(result.msg.encode("utf-8").decode("unicode_escape"))
2022-12-01T11:13:32.5154824Z RuntimeError: On WorkerInfo(id=1, name=worker1):
2022-12-01T11:13:32.5155225Z RuntimeError('CUDA error: invalid device ordinal
2022-12-01T11:13:32.5184489Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-12-01T11:13:32.5184979Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-12-01T11:13:32.5185506Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-12-01T11:13:32.5186472Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f454ee9283b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5187523Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f454ee8e20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5188766Z frame #2: c10::cuda::c10_cuda_check_implementation(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) + 0x42e (0x7f45588f3f1e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-12-01T11:13:32.5189655Z frame #3: <unknown function> + 0x1ab8b (0x7f45588d4b8b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-12-01T11:13:32.5190341Z frame #4: <unknown function> + 0x10c39a8 (0x7f45501989a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5191040Z frame #5: <unknown function> + 0x28e2ed5 (0x7f45519b7ed5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5191741Z frame #6: <unknown function> + 0x28e307b (0x7f45519b807b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5192818Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f455a7aec33 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5193735Z frame #8: <unknown function> + 0x1fbe7e5 (0x7f455aac97e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5194896Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f455a7ebcf8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5195789Z frame #10: <unknown function> + 0x11da60f (0x7f4559ce560f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5196883Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1326 (0x7f455a066476 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5197749Z frame #12: <unknown function> + 0x2195d53 (0x7f455aca0d53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5198820Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5199734Z frame #14: <unknown function> + 0x1fbe4a8 (0x7f455aac94a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5200821Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5201738Z frame #16: <unknown function> + 0x331afa1 (0x7f455be25fa1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5202426Z frame #17: <unknown function> + 0x331b54b (0x7f455be2654b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5203428Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f455a5428b1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5204626Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f455a05e13e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5205499Z frame #20: <unknown function> + 0x23653e9 (0x7f455ae703e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5206603Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f455a6aba55 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5207501Z frame #22: <unknown function> + 0x35e10f (0x7f4566d4b10f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5208182Z frame #23: <unknown function> + 0x35e5bb (0x7f4566d4b5bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5208684Z frame #24: <unknown function> + 0x1ddc68 (0x556f49585c68 in /opt/conda/bin/python)
2022-12-01T11:13:32.5209110Z frame #25: <unknown function> + 0x1049f3 (0x556f494ac9f3 in /opt/conda/bin/python)
2022-12-01T11:13:32.5209528Z frame #26: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5209921Z frame #27: <unknown function> + 0x104425 (0x556f494ac425 in /opt/conda/bin/python)
2022-12-01T11:13:32.5210340Z frame #28: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5210756Z frame #29: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)
2022-12-01T11:13:32.5211219Z frame #30: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)
2022-12-01T11:13:32.5211647Z frame #31: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5212126Z frame #32: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)
2022-12-01T11:13:32.5212550Z frame #33: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)
2022-12-01T11:13:32.5212949Z frame #34: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5213366Z frame #35: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)
2022-12-01T11:13:32.5213806Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556f4958b774 in /opt/conda/bin/python)
2022-12-01T11:13:32.5214223Z frame #37: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5214927Z frame #38: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)
2022-12-01T11:13:32.5215539Z frame #39: <unknown function> + 0xa8449f (0x7f456747149f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5216384Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f456746f6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5217444Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f45674729c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5218650Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f4567473093 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5219960Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f455d41ac14 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5221330Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f45674727b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5222278Z frame #45: <unknown function> + 0x4908e23 (0x7f455d413e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5223265Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f455d4149b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5224418Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f455d40eaf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5225260Z frame #48: <unknown function> + 0x49387d2 (0x7f455d4437d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5225985Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f454ee80c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5226527Z frame #50: <unknown function> + 0xdbbf4 (0x7f457eb5bbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-12-01T11:13:32.5227114Z frame #51: <unknown function> + 0x76db (0x7f459f1af6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-12-01T11:13:32.5227632Z frame #52: clone + 0x3f (0x7f459eed861f in /lib/x86_64-linux-gnu/libc.so.6)
2022-12-01T11:13:32.5227970Z ')
2022-12-01T11:13:32.5228340Z Traceback (most recent call last):
2022-12-01T11:13:32.5228891Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-12-01T11:13:32.5229457Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-12-01T11:13:32.5230075Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 91, in _create_module
2022-12-01T11:13:32.5230489Z     module.to(device)
2022-12-01T11:13:32.5230948Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 987, in to
2022-12-01T11:13:32.5231344Z     return self._apply(convert)
2022-12-01T11:13:32.5231854Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 662, in _apply
2022-12-01T11:13:32.5232234Z     param_applied = fn(param)
2022-12-01T11:13:32.5232744Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 985, in convert
2022-12-01T11:13:32.5233248Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-12-01T11:13:32.5233654Z RuntimeError: CUDA error: invalid device ordinal
2022-12-01T11:13:32.5234134Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-12-01T11:13:32.5234611Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-12-01T11:13:32.5235133Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-12-01T11:13:32.5236032Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f454ee9283b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5237072Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f454ee8e20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5238304Z frame #2: c10::cuda::c10_cuda_check_implementation(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, bool) + 0x42e (0x7f45588f3f1e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-12-01T11:13:32.5239208Z frame #3: <unknown function> + 0x1ab8b (0x7f45588d4b8b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-12-01T11:13:32.5239893Z frame #4: <unknown function> + 0x10c39a8 (0x7f45501989a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5240565Z frame #5: <unknown function> + 0x28e2ed5 (0x7f45519b7ed5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5241266Z frame #6: <unknown function> + 0x28e307b (0x7f45519b807b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.5242363Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f455a7aec33 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5243279Z frame #8: <unknown function> + 0x1fbe7e5 (0x7f455aac97e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5244311Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f455a7ebcf8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5245168Z frame #10: <unknown function> + 0x11da60f (0x7f4559ce560f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5246228Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1326 (0x7f455a066476 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5247142Z frame #12: <unknown function> + 0x2195d53 (0x7f455aca0d53 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5248231Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5249151Z frame #14: <unknown function> + 0x1fbe4a8 (0x7f455aac94a8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5250228Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f455a4ea7a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5251146Z frame #16: <unknown function> + 0x331afa1 (0x7f455be25fa1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5251834Z frame #17: <unknown function> + 0x331b54b (0x7f455be2654b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5252853Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f455a5428b1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5254043Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f455a05e13e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5255186Z frame #20: <unknown function> + 0x23653e9 (0x7f455ae703e9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5256193Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f455a6aba55 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5257093Z frame #22: <unknown function> + 0x35e10f (0x7f4566d4b10f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5257791Z frame #23: <unknown function> + 0x35e5bb (0x7f4566d4b5bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5258290Z frame #24: <unknown function> + 0x1ddc68 (0x556f49585c68 in /opt/conda/bin/python)
2022-12-01T11:13:32.5258701Z frame #25: <unknown function> + 0x1049f3 (0x556f494ac9f3 in /opt/conda/bin/python)
2022-12-01T11:13:32.5259119Z frame #26: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5259535Z frame #27: <unknown function> + 0x104425 (0x556f494ac425 in /opt/conda/bin/python)
2022-12-01T11:13:32.5259951Z frame #28: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5260352Z frame #29: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)
2022-12-01T11:13:32.5260771Z frame #30: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)
2022-12-01T11:13:32.5261191Z frame #31: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5261668Z frame #32: <unknown function> + 0x18fc9b (0x556f49537c9b in /opt/conda/bin/python)
2022-12-01T11:13:32.5262106Z frame #33: <unknown function> + 0x1052a5 (0x556f494ad2a5 in /opt/conda/bin/python)
2022-12-01T11:13:32.5262524Z frame #34: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5263004Z frame #35: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)
2022-12-01T11:13:32.5263428Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x556f4958b774 in /opt/conda/bin/python)
2022-12-01T11:13:32.5263862Z frame #37: <unknown function> + 0x18f742 (0x556f49537742 in /opt/conda/bin/python)
2022-12-01T11:13:32.5264279Z frame #38: _PyObject_Call + 0x20a (0x556f494effaa in /opt/conda/bin/python)
2022-12-01T11:13:32.5264896Z frame #39: <unknown function> + 0xa8449f (0x7f456747149f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5265734Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f456746f6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5266805Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f45674729c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5268005Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f4567473093 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5269307Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f455d41ac14 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5270671Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f45674727b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.5271608Z frame #45: <unknown function> + 0x4908e23 (0x7f455d413e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5272599Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f455d4149b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5273746Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f455d40eaf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5274596Z frame #48: <unknown function> + 0x49387d2 (0x7f455d4437d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.5275329Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f454ee80c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.5275856Z frame #50: <unknown function> + 0xdbbf4 (0x7f457eb5bbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-12-01T11:13:32.5276451Z frame #51: <unknown function> + 0x76db (0x7f459f1af6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-12-01T11:13:32.5276989Z frame #52: clone + 0x3f (0x7f459eed861f in /lib/x86_64-linux-gnu/libc.so.6)
2022-12-01T11:13:32.5277233Z 
2022-12-01T11:13:32.5277253Z 
2022-12-01T11:13:32.5277272Z 
2022-12-01T11:13:32.5277378Z ok (4.085s)
2022-12-01T11:13:32.5277533Z 
2022-12-01T11:13:32.5277798Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5278148Z Ran 1 test in 4.085s
2022-12-01T11:13:32.5278375Z 
2022-12-01T11:13:32.5278474Z OK
2022-12-01T11:13:32.5278618Z 
2022-12-01T11:13:32.5278728Z Generating XML reports...
2022-12-01T11:13:32.5279491Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105530.xml
2022-12-01T11:13:32.5280304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5280785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5281378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5281881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5282374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7da6v1h
2022-12-01T11:13:32.5282945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7da6v1h/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5283246Z 
2022-12-01T11:13:32.5283360Z Running tests...
2022-12-01T11:13:32.5283789Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5284405Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5284967Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5285493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39877
2022-12-01T11:13:32.5285976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39878
2022-12-01T11:13:32.5286629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5287093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5287711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5288239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5288827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5289305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5289920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5290426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5290898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0ovy2hh
2022-12-01T11:13:32.5291468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0ovy2hh/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5292030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1hqn5ef5
2022-12-01T11:13:32.5292597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1hqn5ef5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5293115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5293624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5294044Z fi_getinfo: -61
2022-12-01T11:13:32.5294682Z fi_getinfo: -61
2022-12-01T11:13:32.5295084Z ok (5.482s)
2022-12-01T11:13:32.5295225Z 
2022-12-01T11:13:32.5295483Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5295797Z Ran 1 test in 5.483s
2022-12-01T11:13:32.5295959Z 
2022-12-01T11:13:32.5296051Z OK
2022-12-01T11:13:32.5296184Z 
2022-12-01T11:13:32.5296310Z Generating XML reports...
2022-12-01T11:13:32.5296990Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105536.xml
2022-12-01T11:13:32.5297838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5298309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5298956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5299416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5299886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjyfj2j4
2022-12-01T11:13:32.5300434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjyfj2j4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5300738Z 
2022-12-01T11:13:32.5300848Z Running tests...
2022-12-01T11:13:32.5301234Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5301814Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5302360Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5302853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40067
2022-12-01T11:13:32.5303292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40068
2022-12-01T11:13:32.5303739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 40069
2022-12-01T11:13:32.5304190Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 40070
2022-12-01T11:13:32.5304783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5305233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5305811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5306330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5306896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5307350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5307921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5308369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5308945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5309392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5309964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5310411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5310988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5311438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5311992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5312457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5312921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjs0200m
2022-12-01T11:13:32.5313465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjs0200m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5313983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6vp98yh
2022-12-01T11:13:32.5314523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6vp98yh/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5315111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59js2tte
2022-12-01T11:13:32.5315655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59js2tte/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5316216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdg0murs
2022-12-01T11:13:32.5316755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdg0murs/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5317266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5317723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5318192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5318657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5319050Z fi_getinfo: -61
2022-12-01T11:13:32.5319311Z fi_getinfo: -61
2022-12-01T11:13:32.5319583Z fi_getinfo: -61
2022-12-01T11:13:32.5319856Z fi_getinfo: -61
2022-12-01T11:13:32.5320071Z ok (8.298s)
2022-12-01T11:13:32.5320223Z 
2022-12-01T11:13:32.5320494Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5320821Z Ran 1 test in 8.298s
2022-12-01T11:13:32.5320984Z 
2022-12-01T11:13:32.5321057Z OK
2022-12-01T11:13:32.5321190Z 
2022-12-01T11:13:32.5321314Z Generating XML reports...
2022-12-01T11:13:32.5321951Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221201105544.xml
2022-12-01T11:13:32.5322693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5323130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5323706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5324182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5324640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoxi_5po
2022-12-01T11:13:32.5325185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoxi_5po/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5325484Z 
2022-12-01T11:13:32.5325594Z Running tests...
2022-12-01T11:13:32.5325998Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5326553Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5327106Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5327612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40414
2022-12-01T11:13:32.5328050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40415
2022-12-01T11:13:32.5328665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5329121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5329695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5330149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5330729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5331176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5331750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5332199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5332716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5sc5m9iw
2022-12-01T11:13:32.5333265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5sc5m9iw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5333825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph06bt93u
2022-12-01T11:13:32.5334615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph06bt93u/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5335121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5335587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5335964Z skip: Need at least 4 CUDA devices (3.781s)
2022-12-01T11:13:32.5336156Z 
2022-12-01T11:13:32.5336436Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5336766Z Ran 1 test in 3.781s
2022-12-01T11:13:32.5336927Z 
2022-12-01T11:13:32.5337018Z OK (skipped=1)
2022-12-01T11:13:32.5337180Z 
2022-12-01T11:13:32.5337305Z Generating XML reports...
2022-12-01T11:13:32.5337967Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105554.xml
2022-12-01T11:13:32.5338730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5339164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5339738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5340211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5340660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7nqy2gc
2022-12-01T11:13:32.5341205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7nqy2gc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5341507Z 
2022-12-01T11:13:32.5341620Z Running tests...
2022-12-01T11:13:32.5342026Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5342591Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5343154Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5343659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40517
2022-12-01T11:13:32.5344117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40518
2022-12-01T11:13:32.5344706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5345159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5345735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5346193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5346768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5347220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5347793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5348242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5348706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yns_5nv
2022-12-01T11:13:32.5349243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yns_5nv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5349782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2kyhmvq7
2022-12-01T11:13:32.5350402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2kyhmvq7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5350927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5351479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5351858Z skip: Need at least 4 CUDA devices (3.802s)
2022-12-01T11:13:32.5352052Z 
2022-12-01T11:13:32.5352324Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5352653Z Ran 1 test in 3.802s
2022-12-01T11:13:32.5352813Z 
2022-12-01T11:13:32.5352902Z OK (skipped=1)
2022-12-01T11:13:32.5353056Z 
2022-12-01T11:13:32.5353179Z Generating XML reports...
2022-12-01T11:13:32.5353835Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105600.xml
2022-12-01T11:13:32.5354595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5355030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5355606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5356080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5356551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ogvumu7
2022-12-01T11:13:32.5357076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ogvumu7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5357381Z 
2022-12-01T11:13:32.5357490Z Running tests...
2022-12-01T11:13:32.5357897Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5358456Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5359012Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5359508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40620
2022-12-01T11:13:32.5359969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40621
2022-12-01T11:13:32.5360558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5361004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5361580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5362033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5362614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5363059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5363634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5364084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5364554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiqyu3v6
2022-12-01T11:13:32.5365098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiqyu3v6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5365688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2pbq1jf
2022-12-01T11:13:32.5366206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2pbq1jf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5366720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5367196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5367570Z skip: Need at least 4 CUDA devices (3.794s)
2022-12-01T11:13:32.5367820Z 
2022-12-01T11:13:32.5368100Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5368427Z Ran 1 test in 3.794s
2022-12-01T11:13:32.5368635Z 
2022-12-01T11:13:32.5368745Z OK (skipped=1)
2022-12-01T11:13:32.5368881Z 
2022-12-01T11:13:32.5369011Z Generating XML reports...
2022-12-01T11:13:32.5369665Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105606.xml
2022-12-01T11:13:32.5370421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5370857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5371431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5371904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5372374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6aipb0bf
2022-12-01T11:13:32.5372895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6aipb0bf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5373201Z 
2022-12-01T11:13:32.5373312Z Running tests...
2022-12-01T11:13:32.5373714Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5374275Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5375100Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5375579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40723
2022-12-01T11:13:32.5376041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40724
2022-12-01T11:13:32.5376645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5377096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5377675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5378151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5378714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5379161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5379730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5380175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5380644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2rncgaw
2022-12-01T11:13:32.5381194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2rncgaw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5381732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0su0x3j
2022-12-01T11:13:32.5382256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0su0x3j/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5382767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5383240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5383615Z skip: Need at least 4 CUDA devices (3.783s)
2022-12-01T11:13:32.5383810Z 
2022-12-01T11:13:32.5384081Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5384409Z Ran 1 test in 3.783s
2022-12-01T11:13:32.5384570Z 
2022-12-01T11:13:32.5384679Z OK (skipped=1)
2022-12-01T11:13:32.5384815Z 
2022-12-01T11:13:32.5384939Z Generating XML reports...
2022-12-01T11:13:32.5385674Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105612.xml
2022-12-01T11:13:32.5386443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5386939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5387521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5387992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5388459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01gft8r0
2022-12-01T11:13:32.5388981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01gft8r0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5389281Z 
2022-12-01T11:13:32.5389392Z Running tests...
2022-12-01T11:13:32.5389803Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5390385Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5390965Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5391467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40826
2022-12-01T11:13:32.5391884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40827
2022-12-01T11:13:32.5392478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5392930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5393503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5393972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5394535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5394985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5395560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5396009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5396474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp65ac3p92
2022-12-01T11:13:32.5397019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp65ac3p92/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5397554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5i0pwz66
2022-12-01T11:13:32.5398071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5i0pwz66/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5398584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5399057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5399456Z skip: Need at least 4 CUDA devices (3.805s)
2022-12-01T11:13:32.5399631Z 
2022-12-01T11:13:32.5399902Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5400229Z Ran 1 test in 3.805s
2022-12-01T11:13:32.5400389Z 
2022-12-01T11:13:32.5400498Z OK (skipped=1)
2022-12-01T11:13:32.5400652Z 
2022-12-01T11:13:32.5400757Z Generating XML reports...
2022-12-01T11:13:32.5401413Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105618.xml
2022-12-01T11:13:32.5402168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5402623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5403238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5403717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5404231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfd8gacb
2022-12-01T11:13:32.5404757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfd8gacb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5405062Z 
2022-12-01T11:13:32.5405170Z Running tests...
2022-12-01T11:13:32.5405575Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5406149Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5406737Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5407249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40929
2022-12-01T11:13:32.5407705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40930
2022-12-01T11:13:32.5408301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5408753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5409327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5409799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5410359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5410807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5411380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5411848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5412293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_n07ct_v
2022-12-01T11:13:32.5412836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_n07ct_v/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5413368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwm7yr93
2022-12-01T11:13:32.5413888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwm7yr93/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5414657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5415150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5415519Z skip: Need at least 4 CUDA devices (3.810s)
2022-12-01T11:13:32.5415695Z 
2022-12-01T11:13:32.5415979Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5416307Z Ran 1 test in 3.810s
2022-12-01T11:13:32.5416468Z 
2022-12-01T11:13:32.5416578Z OK (skipped=1)
2022-12-01T11:13:32.5416734Z 
2022-12-01T11:13:32.5416843Z Generating XML reports...
2022-12-01T11:13:32.5417504Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105624.xml
2022-12-01T11:13:32.5418258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5418717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5419274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5419746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5420212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwkj12oy7
2022-12-01T11:13:32.5420817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwkj12oy7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5421133Z 
2022-12-01T11:13:32.5421243Z Running tests...
2022-12-01T11:13:32.5421710Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5422288Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5422821Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5423319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41032
2022-12-01T11:13:32.5423770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41033
2022-12-01T11:13:32.5424401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5424813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5425397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5425870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5426439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5426885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5427452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5427918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5428372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyz1aaqk
2022-12-01T11:13:32.5428917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyz1aaqk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5429454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s4044j4
2022-12-01T11:13:32.5429971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s4044j4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5430480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5430952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5431347Z skip: Need at least 4 CUDA devices (3.800s)
2022-12-01T11:13:32.5431538Z 
2022-12-01T11:13:32.5431792Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5432116Z Ran 1 test in 3.800s
2022-12-01T11:13:32.5432276Z 
2022-12-01T11:13:32.5432385Z OK (skipped=1)
2022-12-01T11:13:32.5432539Z 
2022-12-01T11:13:32.5432652Z Generating XML reports...
2022-12-01T11:13:32.5433300Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105630.xml
2022-12-01T11:13:32.5434073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5434515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5435071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5435541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5436011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzi1aczdn
2022-12-01T11:13:32.5436560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzi1aczdn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5436840Z 
2022-12-01T11:13:32.5436948Z Running tests...
2022-12-01T11:13:32.5437349Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5437969Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5438524Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5439092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41135
2022-12-01T11:13:32.5439548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41136
2022-12-01T11:13:32.5440160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5440597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5441177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5441650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5442216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5442661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5443232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5443704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5444151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsskoyeqr
2022-12-01T11:13:32.5444696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsskoyeqr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5445232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4z3_q5ur
2022-12-01T11:13:32.5445764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4z3_q5ur/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5446253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5446729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5447128Z skip: Need at least 4 CUDA devices (3.786s)
2022-12-01T11:13:32.5447329Z 
2022-12-01T11:13:32.5447581Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5447909Z Ran 1 test in 3.787s
2022-12-01T11:13:32.5448069Z 
2022-12-01T11:13:32.5448178Z OK (skipped=1)
2022-12-01T11:13:32.5448334Z 
2022-12-01T11:13:32.5448459Z Generating XML reports...
2022-12-01T11:13:32.5449096Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105636.xml
2022-12-01T11:13:32.5449852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5450302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5450863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5451337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5451800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm_o9m52
2022-12-01T11:13:32.5452341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm_o9m52/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5452638Z 
2022-12-01T11:13:32.5452730Z Running tests...
2022-12-01T11:13:32.5453134Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5453710Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5454301Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5455126Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41238
2022-12-01T11:13:32.5455637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41239
2022-12-01T11:13:32.5456099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41240
2022-12-01T11:13:32.5456594Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41241
2022-12-01T11:13:32.5457211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5457664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5458232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5458664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5459241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5459712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5460288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5460759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5461341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5461789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5462340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5462803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5463380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5463807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5464383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5464846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5465316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld29nk_u
2022-12-01T11:13:32.5465841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld29nk_u/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5466377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j0ue0rm
2022-12-01T11:13:32.5466917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j0ue0rm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5467450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvn7c36u
2022-12-01T11:13:32.5467971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvn7c36u/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5468509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbzo4uqp
2022-12-01T11:13:32.5469050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbzo4uqp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5469552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5470026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5470484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5470956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5471327Z fi_getinfo: -61
2022-12-01T11:13:32.5471609Z fi_getinfo: -61
2022-12-01T11:13:32.5471881Z fi_getinfo: -61
2022-12-01T11:13:32.5472132Z fi_getinfo: -61
2022-12-01T11:13:32.5472360Z ok (11.322s)
2022-12-01T11:13:32.5472508Z 
2022-12-01T11:13:32.5472775Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5473088Z Ran 1 test in 11.322s
2022-12-01T11:13:32.5473304Z 
2022-12-01T11:13:32.5473403Z OK
2022-12-01T11:13:32.5473422Z 
2022-12-01T11:13:32.5473549Z Generating XML reports...
2022-12-01T11:13:32.5474166Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105642.xml
2022-12-01T11:13:32.5474539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5474717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5475104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5475277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5475536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp480bjgl3
2022-12-01T11:13:32.5475811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp480bjgl3/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5475832Z 
2022-12-01T11:13:32.5475941Z Running tests...
2022-12-01T11:13:32.5476204Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5476568Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5476888Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5477110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41585
2022-12-01T11:13:32.5477310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41586
2022-12-01T11:13:32.5477528Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41587
2022-12-01T11:13:32.5477742Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41588
2022-12-01T11:13:32.5478119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5478297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5478683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5478874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5479245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5479419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5479775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5479964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5480331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5480506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5480931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5481071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5481436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5481609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5481967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5482154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5482411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpozebg1yn
2022-12-01T11:13:32.5482733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpozebg1yn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5482999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8twhds2
2022-12-01T11:13:32.5483354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8twhds2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5483609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9fv9jdt
2022-12-01T11:13:32.5483878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9fv9jdt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5484130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbz7u69u2
2022-12-01T11:13:32.5484376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbz7u69u2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5484607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5484836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5485064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5485292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5485441Z fi_getinfo: -61
2022-12-01T11:13:32.5485578Z fi_getinfo: -61
2022-12-01T11:13:32.5485714Z fi_getinfo: -61
2022-12-01T11:13:32.5485834Z fi_getinfo: -61
2022-12-01T11:13:32.5485934Z ok (11.196s)
2022-12-01T11:13:32.5485954Z 
2022-12-01T11:13:32.5486219Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5486333Z Ran 1 test in 11.197s
2022-12-01T11:13:32.5486352Z 
2022-12-01T11:13:32.5486445Z OK
2022-12-01T11:13:32.5486465Z 
2022-12-01T11:13:32.5486590Z Generating XML reports...
2022-12-01T11:13:32.5487144Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105656.xml
2022-12-01T11:13:32.5487520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5487684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5488068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5488259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5488514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2jnr3k74
2022-12-01T11:13:32.5488780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2jnr3k74/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5488800Z 
2022-12-01T11:13:32.5488910Z Running tests...
2022-12-01T11:13:32.5489173Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5489534Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5489843Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5490069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41932
2022-12-01T11:13:32.5490284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41933
2022-12-01T11:13:32.5490499Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 41934
2022-12-01T11:13:32.5490711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 41935
2022-12-01T11:13:32.5491084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5491262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5491643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5491899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5492262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5492484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5492859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5493049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5493412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5493588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5493961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5494152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5494745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5494943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5495292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5495479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5495742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3ve8m2l
2022-12-01T11:13:32.5496014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3ve8m2l/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5496269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmik3bdri
2022-12-01T11:13:32.5496547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmik3bdri/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5496801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprep2d1jh
2022-12-01T11:13:32.5497056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprep2d1jh/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5497311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0kt8n18r
2022-12-01T11:13:32.5497578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0kt8n18r/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5497809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5498034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5498261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5498489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5498596Z ok (10.607s)
2022-12-01T11:13:32.5498616Z 
2022-12-01T11:13:32.5498870Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5498988Z Ran 1 test in 10.607s
2022-12-01T11:13:32.5499007Z 
2022-12-01T11:13:32.5499102Z OK
2022-12-01T11:13:32.5499121Z 
2022-12-01T11:13:32.5499247Z Generating XML reports...
2022-12-01T11:13:32.5499798Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105709.xml
2022-12-01T11:13:32.5500170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5500348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5500728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5500922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5501243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3ncr3t0
2022-12-01T11:13:32.5501524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3ncr3t0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5501596Z 
2022-12-01T11:13:32.5501710Z Running tests...
2022-12-01T11:13:32.5501977Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5502340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5502675Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5502896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42111
2022-12-01T11:13:32.5503119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42112
2022-12-01T11:13:32.5503340Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42113
2022-12-01T11:13:32.5503539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42114
2022-12-01T11:13:32.5503918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5504098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5504482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5504673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5505041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5505218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5505599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5505775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5506141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5506317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5506732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5506921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5507284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5507456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5507832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5508022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5508264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9rctdyk
2022-12-01T11:13:32.5508543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9rctdyk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5508797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhxsa8kl
2022-12-01T11:13:32.5509067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhxsa8kl/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5509321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1keb9ma
2022-12-01T11:13:32.5509587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1keb9ma/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5509840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqiltravf
2022-12-01T11:13:32.5510162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqiltravf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5510381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5510648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5510876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5511102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5511204Z ok (9.391s)
2022-12-01T11:13:32.5511224Z 
2022-12-01T11:13:32.5511496Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5511608Z Ran 1 test in 9.391s
2022-12-01T11:13:32.5511628Z 
2022-12-01T11:13:32.5511722Z OK
2022-12-01T11:13:32.5511741Z 
2022-12-01T11:13:32.5511865Z Generating XML reports...
2022-12-01T11:13:32.5512399Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105722.xml
2022-12-01T11:13:32.5512773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5512954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5513335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5513526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5513781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzbxodh1u
2022-12-01T11:13:32.5514055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzbxodh1u/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5514075Z 
2022-12-01T11:13:32.5514183Z Running tests...
2022-12-01T11:13:32.5514430Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5514796Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5515122Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5515348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42346
2022-12-01T11:13:32.5515566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42347
2022-12-01T11:13:32.5515781Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42348
2022-12-01T11:13:32.5515995Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42349
2022-12-01T11:13:32.5516369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5516546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5516911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5517100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5517471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5517650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5518023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5518211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5518576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5518751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5519105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5519343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5519712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5519930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5520301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5520493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5520752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7p0yzvzu
2022-12-01T11:13:32.5521023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7p0yzvzu/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5521275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoe354fn6
2022-12-01T11:13:32.5521527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoe354fn6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5521783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzummbwb
2022-12-01T11:13:32.5522056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzummbwb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5522310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8ssfe58
2022-12-01T11:13:32.5522578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8ssfe58/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5522806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5523028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5523254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5523484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5523569Z ok (9.375s)
2022-12-01T11:13:32.5523589Z 
2022-12-01T11:13:32.5523860Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5523976Z Ran 1 test in 9.375s
2022-12-01T11:13:32.5523996Z 
2022-12-01T11:13:32.5524089Z OK
2022-12-01T11:13:32.5524108Z 
2022-12-01T11:13:32.5524232Z Generating XML reports...
2022-12-01T11:13:32.5524783Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105734.xml
2022-12-01T11:13:32.5525156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5525333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5525697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5525895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5526151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhx9pjc6
2022-12-01T11:13:32.5526427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhx9pjc6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5526446Z 
2022-12-01T11:13:32.5526555Z Running tests...
2022-12-01T11:13:32.5526821Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5527183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5527537Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5527758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42521
2022-12-01T11:13:32.5527957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42522
2022-12-01T11:13:32.5528222Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42523
2022-12-01T11:13:32.5528445Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42524
2022-12-01T11:13:32.5528872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5529049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5529430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5529622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5529986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5530140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5530519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5530707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5531074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5531250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5531621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5531808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5532168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5532342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5532704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5532894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5533151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppe2n4rzi
2022-12-01T11:13:32.5533429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppe2n4rzi/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5533684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6got_pm
2022-12-01T11:13:32.5533952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6got_pm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5534208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk97vg1_g
2022-12-01T11:13:32.5534716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk97vg1_g/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5534953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpchwa52vr
2022-12-01T11:13:32.5535203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpchwa52vr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5535430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5535660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5535887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5536113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5536216Z ok (9.395s)
2022-12-01T11:13:32.5536235Z 
2022-12-01T11:13:32.5536513Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5536630Z Ran 1 test in 9.395s
2022-12-01T11:13:32.5536650Z 
2022-12-01T11:13:32.5536723Z OK
2022-12-01T11:13:32.5536742Z 
2022-12-01T11:13:32.5536866Z Generating XML reports...
2022-12-01T11:13:32.5537496Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105745.xml
2022-12-01T11:13:32.5537885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5538137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5538519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5538711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5538969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlzz59td
2022-12-01T11:13:32.5539220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlzz59td/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5539260Z 
2022-12-01T11:13:32.5539352Z Running tests...
2022-12-01T11:13:32.5539617Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5539982Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5540328Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5540552Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42760
2022-12-01T11:13:32.5540769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42761
2022-12-01T11:13:32.5540985Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42762
2022-12-01T11:13:32.5541199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42763
2022-12-01T11:13:32.5541552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5541731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5542117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5542314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5542686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5542860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5543239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5543432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5543795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5543952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5544327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5544516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5544882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5545055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5545431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5545621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5545879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xn16ydy
2022-12-01T11:13:32.5546133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xn16ydy/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5546389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvstf5ebf
2022-12-01T11:13:32.5546712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvstf5ebf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5546976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwatnvmnv
2022-12-01T11:13:32.5547290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwatnvmnv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5547542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvud7esv
2022-12-01T11:13:32.5547812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvud7esv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5548046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5548269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5548478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5548709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5548812Z ok (9.406s)
2022-12-01T11:13:32.5548832Z 
2022-12-01T11:13:32.5549107Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5549220Z Ran 1 test in 9.406s
2022-12-01T11:13:32.5549240Z 
2022-12-01T11:13:32.5549333Z OK
2022-12-01T11:13:32.5549352Z 
2022-12-01T11:13:32.5549476Z Generating XML reports...
2022-12-01T11:13:32.5550026Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105757.xml
2022-12-01T11:13:32.5550381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5550559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5550940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5551135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5551387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89t2l1wk
2022-12-01T11:13:32.5551661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89t2l1wk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5551681Z 
2022-12-01T11:13:32.5551790Z Running tests...
2022-12-01T11:13:32.5552057Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5552418Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5552743Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5552963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42939
2022-12-01T11:13:32.5553186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42940
2022-12-01T11:13:32.5553403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 42941
2022-12-01T11:13:32.5553616Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 42942
2022-12-01T11:13:32.5553998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5554176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5554578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5554753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5555127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5555313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5555735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5555926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5556341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5556525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5556894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5557084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5557428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5557604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5557987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5558180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5558444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnzxd_q4
2022-12-01T11:13:32.5558716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnzxd_q4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5558974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcfxc512
2022-12-01T11:13:32.5559247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcfxc512/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5559495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdkkpv5k
2022-12-01T11:13:32.5559759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdkkpv5k/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5560174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl45gbb_2
2022-12-01T11:13:32.5560868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl45gbb_2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5561435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5561992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5562545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5563117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5563348Z ok (9.420s)
2022-12-01T11:13:32.5563379Z 
2022-12-01T11:13:32.5563986Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5564225Z Ran 1 test in 9.420s
2022-12-01T11:13:32.5564269Z 
2022-12-01T11:13:32.5564480Z OK
2022-12-01T11:13:32.5564507Z 
2022-12-01T11:13:32.5564797Z Generating XML reports...
2022-12-01T11:13:32.5566174Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105809.xml
2022-12-01T11:13:32.5567105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5567536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5568483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5568953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5569543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zlr2da8
2022-12-01T11:13:32.5570195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zlr2da8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5570236Z 
2022-12-01T11:13:32.5570489Z Running tests...
2022-12-01T11:13:32.5571209Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5572128Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5573025Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5573557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43174
2022-12-01T11:13:32.5574052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43175
2022-12-01T11:13:32.5575036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43176
2022-12-01T11:13:32.5575575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43177
2022-12-01T11:13:32.5576544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5576968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5577953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5578415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5579352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5579774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5580706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5581136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5582030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5582444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5583374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5583812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5584721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5585148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5586080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5586508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5587136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi3nwzivs
2022-12-01T11:13:32.5587804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi3nwzivs/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5588420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4nzz6me
2022-12-01T11:13:32.5589082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4nzz6me/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5589701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14ev_d9r
2022-12-01T11:13:32.5590334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14ev_d9r/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5590951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps8o1vacn
2022-12-01T11:13:32.5591616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps8o1vacn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5592171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5592716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5593262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5593972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5594205Z ok (9.407s)
2022-12-01T11:13:32.5594339Z 
2022-12-01T11:13:32.5594991Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5595245Z Ran 1 test in 9.407s
2022-12-01T11:13:32.5595275Z 
2022-12-01T11:13:32.5595482Z OK
2022-12-01T11:13:32.5595515Z 
2022-12-01T11:13:32.5595762Z Generating XML reports...
2022-12-01T11:13:32.5597149Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105820.xml
2022-12-01T11:13:32.5598065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5598489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5599450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5599918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5600549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsuujzd7
2022-12-01T11:13:32.5601214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsuujzd7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5601251Z 
2022-12-01T11:13:32.5601505Z Running tests...
2022-12-01T11:13:32.5602095Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5602991Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5603753Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5604293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43349
2022-12-01T11:13:32.5604831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43350
2022-12-01T11:13:32.5605357Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43351
2022-12-01T11:13:32.5605883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43352
2022-12-01T11:13:32.5606852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5607237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5608177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5608643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5609554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5609976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5610926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5611383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5612306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5612716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5613630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5614086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5615464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5615894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5617012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5617479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5618234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpaxbpj5
2022-12-01T11:13:32.5618935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpaxbpj5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5619540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6up1hqky
2022-12-01T11:13:32.5620214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6up1hqky/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5620863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywqhmm1g
2022-12-01T11:13:32.5621533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywqhmm1g/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5622164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpata2gzzp
2022-12-01T11:13:32.5622825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpata2gzzp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5623410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5623977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5624539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5625080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5625307Z ok (3.876s)
2022-12-01T11:13:32.5625350Z 
2022-12-01T11:13:32.5625990Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5626250Z Ran 1 test in 3.876s
2022-12-01T11:13:32.5626283Z 
2022-12-01T11:13:32.5626504Z OK
2022-12-01T11:13:32.5626544Z 
2022-12-01T11:13:32.5626835Z Generating XML reports...
2022-12-01T11:13:32.5628237Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105832.xml
2022-12-01T11:13:32.5629190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5629614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5630548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5631019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5631659Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyoq5bz1
2022-12-01T11:13:32.5632341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyoq5bz1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5632377Z 
2022-12-01T11:13:32.5632633Z Running tests...
2022-12-01T11:13:32.5633265Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5634178Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5634931Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5635465Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43520
2022-12-01T11:13:32.5635998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43521
2022-12-01T11:13:32.5636539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43522
2022-12-01T11:13:32.5637072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43523
2022-12-01T11:13:32.5638009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5638431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5639501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5639988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5641014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5641403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5642354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5642841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5643793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5644240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5645207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5645701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5646668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5647083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5648068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5648541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5649218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj745q5st
2022-12-01T11:13:32.5649893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj745q5st/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5650570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlj92yvk
2022-12-01T11:13:32.5651262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlj92yvk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5651943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13zhct7e
2022-12-01T11:13:32.5652634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13zhct7e/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5653272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93pt3vr0
2022-12-01T11:13:32.5653970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93pt3vr0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5655117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5655737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5656327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5656926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5657174Z ok (3.786s)
2022-12-01T11:13:32.5657222Z 
2022-12-01T11:13:32.5657893Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5658103Z Ran 1 test in 3.787s
2022-12-01T11:13:32.5658198Z 
2022-12-01T11:13:32.5658363Z OK
2022-12-01T11:13:32.5658394Z 
2022-12-01T11:13:32.5658679Z Generating XML reports...
2022-12-01T11:13:32.5660130Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105838.xml
2022-12-01T11:13:32.5661082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5661521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5662658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5663164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5663833Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm6qfr0xd
2022-12-01T11:13:32.5664652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm6qfr0xd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5664702Z 
2022-12-01T11:13:32.5664958Z Running tests...
2022-12-01T11:13:32.5665701Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5666660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5667474Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5668037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43691
2022-12-01T11:13:32.5668607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43692
2022-12-01T11:13:32.5669150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43693
2022-12-01T11:13:32.5669678Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43694
2022-12-01T11:13:32.5670667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5671112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5672074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5672526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5673511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5673964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5681730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5682226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5683227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5683716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5684734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5685228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5686190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5686626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5687633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5688115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5688793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvmi74_2
2022-12-01T11:13:32.5689489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvmi74_2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5690154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9sk4jvrn
2022-12-01T11:13:32.5690864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9sk4jvrn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5691524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp546h_i92
2022-12-01T11:13:32.5692226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp546h_i92/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5693017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sd9sqz1
2022-12-01T11:13:32.5693733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sd9sqz1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5695163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5695760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5696354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5696944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5697195Z ok (3.893s)
2022-12-01T11:13:32.5697242Z 
2022-12-01T11:13:32.5697940Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5698211Z Ran 1 test in 3.893s
2022-12-01T11:13:32.5698246Z 
2022-12-01T11:13:32.5698462Z OK
2022-12-01T11:13:32.5698508Z 
2022-12-01T11:13:32.5698819Z Generating XML reports...
2022-12-01T11:13:32.5700280Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105844.xml
2022-12-01T11:13:32.5701265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5701712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5702713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5703212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5703861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4e44te2
2022-12-01T11:13:32.5704564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4e44te2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5704596Z 
2022-12-01T11:13:32.5704864Z Running tests...
2022-12-01T11:13:32.5705534Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5706542Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5707325Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5707895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43862
2022-12-01T11:13:32.5708462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43863
2022-12-01T11:13:32.5709053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43864
2022-12-01T11:13:32.5709603Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43865
2022-12-01T11:13:32.5710603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5711049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5712041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5712451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5713475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5713954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5714968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5715450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5716440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5716878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5718108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5718626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5719686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5720133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5721123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5721622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5722301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfed0mgof
2022-12-01T11:13:32.5722957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkl1j6i8
2022-12-01T11:13:32.5723614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_cbn3b5y
2022-12-01T11:13:32.5724321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfed0mgof/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5724984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkl1j6i8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5725678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_cbn3b5y/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5726316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jyqorfz
2022-12-01T11:13:32.5727033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jyqorfz/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5727639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5728235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5728837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5729442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5729675Z ok (3.894s)
2022-12-01T11:13:32.5729718Z 
2022-12-01T11:13:32.5730371Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5730636Z Ran 1 test in 3.894s
2022-12-01T11:13:32.5730671Z 
2022-12-01T11:13:32.5730905Z OK
2022-12-01T11:13:32.5730934Z 
2022-12-01T11:13:32.5731242Z Generating XML reports...
2022-12-01T11:13:32.5732692Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105850.xml
2022-12-01T11:13:32.5733699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5734158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5735538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5736028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5736685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ex0kz9g
2022-12-01T11:13:32.5737394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ex0kz9g/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5737437Z 
2022-12-01T11:13:32.5737698Z Running tests...
2022-12-01T11:13:32.5738372Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5739312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5740152Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5740729Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44033
2022-12-01T11:13:32.5741424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44034
2022-12-01T11:13:32.5741972Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44035
2022-12-01T11:13:32.5742650Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44036
2022-12-01T11:13:32.5743641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5744095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5745117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5745607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5746587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5747027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5748010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5748485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5749464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5749919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5750902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5751380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5752367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5752792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5753811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5754261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5754948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xpqkmpw
2022-12-01T11:13:32.5755638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xpqkmpw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5756312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpply30a8m
2022-12-01T11:13:32.5757018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpply30a8m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5757688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtfoj50r
2022-12-01T11:13:32.5758382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtfoj50r/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5759044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphol7uo_y
2022-12-01T11:13:32.5759704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphol7uo_y/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5760305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5760900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5761501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5762111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5762349Z ok (5.486s)
2022-12-01T11:13:32.5762377Z 
2022-12-01T11:13:32.5763056Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5763328Z Ran 1 test in 5.486s
2022-12-01T11:13:32.5763371Z 
2022-12-01T11:13:32.5763598Z OK
2022-12-01T11:13:32.5763633Z 
2022-12-01T11:13:32.5763911Z Generating XML reports...
2022-12-01T11:13:32.5765462Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105856.xml
2022-12-01T11:13:32.5766541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5767001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5768017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5768520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5769184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbir95gia
2022-12-01T11:13:32.5769883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbir95gia/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5769927Z 
2022-12-01T11:13:32.5770189Z Running tests...
2022-12-01T11:13:32.5770838Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5771782Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5772616Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5773180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44208
2022-12-01T11:13:32.5773744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44209
2022-12-01T11:13:32.5774302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44210
2022-12-01T11:13:32.5775391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44211
2022-12-01T11:13:32.5776400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5776802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5777839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5778354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5779315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5779745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5780760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5781242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5782204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5782662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5783627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5784111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5785091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5785534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5786547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5787024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5787714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw9uz6np
2022-12-01T11:13:32.5788417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw9uz6np/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5789189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9aufemhy
2022-12-01T11:13:32.5789920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9aufemhy/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5790692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoq8i3lb0
2022-12-01T11:13:32.5791383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoq8i3lb0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5792045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqibfbyue
2022-12-01T11:13:32.5792764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqibfbyue/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5793372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5793959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5794565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5795141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5795375Z ok (5.461s)
2022-12-01T11:13:32.5795419Z 
2022-12-01T11:13:32.5796096Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5796371Z Ran 1 test in 5.462s
2022-12-01T11:13:32.5796399Z 
2022-12-01T11:13:32.5796627Z OK
2022-12-01T11:13:32.5796655Z 
2022-12-01T11:13:32.5796964Z Generating XML reports...
2022-12-01T11:13:32.5798431Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105904.xml
2022-12-01T11:13:32.5799404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5799815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5800831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5801329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5801988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphc5qw0c
2022-12-01T11:13:32.5802709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphc5qw0c/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5802754Z 
2022-12-01T11:13:32.5803009Z Running tests...
2022-12-01T11:13:32.5803681Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5804631Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5805472Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5805997Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44383
2022-12-01T11:13:32.5806622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44384
2022-12-01T11:13:32.5807168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44385
2022-12-01T11:13:32.5807757Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44386
2022-12-01T11:13:32.5808746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5809189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5810194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5810697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5811654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5812084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5813195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5813691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5815316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5815770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5816739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5817195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5818200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5818662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5819655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5820139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5820830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnljvglaj
2022-12-01T11:13:32.5821532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnljvglaj/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5822190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwwtwxpq
2022-12-01T11:13:32.5822901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwwtwxpq/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5823571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9iyl190
2022-12-01T11:13:32.5824219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9iyl190/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5824903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhcvkn_4
2022-12-01T11:13:32.5825594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhcvkn_4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5826199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5826788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5827390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5827989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5828236Z ok (10.717s)
2022-12-01T11:13:32.5828270Z 
2022-12-01T11:13:32.5828934Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5829197Z Ran 1 test in 10.718s
2022-12-01T11:13:32.5829228Z 
2022-12-01T11:13:32.5829456Z OK
2022-12-01T11:13:32.5829497Z 
2022-12-01T11:13:32.5829802Z Generating XML reports...
2022-12-01T11:13:32.5831264Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105912.xml
2022-12-01T11:13:32.5832258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5832690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5833697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5834193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5834846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr67euwn
2022-12-01T11:13:32.5835540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr67euwn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5835584Z 
2022-12-01T11:13:32.5835838Z Running tests...
2022-12-01T11:13:32.5836644Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5837617Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5838513Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5840530Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.579s)
2022-12-01T11:13:32.5840577Z 
2022-12-01T11:13:32.5841232Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5841505Z Ran 1 test in 1.579s
2022-12-01T11:13:32.5841533Z 
2022-12-01T11:13:32.5841785Z OK (skipped=1)
2022-12-01T11:13:32.5841819Z 
2022-12-01T11:13:32.5842084Z Generating XML reports...
2022-12-01T11:13:32.5843560Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105925.xml
2022-12-01T11:13:32.5844539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5844971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5845986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5846478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5847136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_n246fi
2022-12-01T11:13:32.5847829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_n246fi/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5847875Z 
2022-12-01T11:13:32.5848126Z Running tests...
2022-12-01T11:13:32.5848764Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5849691Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5850480Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5851040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44596
2022-12-01T11:13:32.5851611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44597
2022-12-01T11:13:32.5852164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44598
2022-12-01T11:13:32.5852726Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44599
2022-12-01T11:13:32.5853713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5854128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5855695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5856186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5857174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5857610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5858586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5859059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5860019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5860476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5861600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5862107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5863188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5863631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5864631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5865118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5865780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7z0idq1t
2022-12-01T11:13:32.5866484Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7z0idq1t/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5867130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6w5pk1v
2022-12-01T11:13:32.5867830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6w5pk1v/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5868505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vvcysgl
2022-12-01T11:13:32.5869195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vvcysgl/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5869859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7ac6me5
2022-12-01T11:13:32.5870570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7ac6me5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5871173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5871767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5872367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5872943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5873296Z fi_getinfo: -61
2022-12-01T11:13:32.5873652Z fi_getinfo: -61
2022-12-01T11:13:32.5873985Z fi_getinfo: -61
2022-12-01T11:13:32.5874320Z fi_getinfo: -61
2022-12-01T11:13:32.5874553Z ok (19.327s)
2022-12-01T11:13:32.5874581Z 
2022-12-01T11:13:32.5875245Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5875487Z Ran 1 test in 19.327s
2022-12-01T11:13:32.5875519Z 
2022-12-01T11:13:32.5875735Z OK
2022-12-01T11:13:32.5875785Z 
2022-12-01T11:13:32.5876084Z Generating XML reports...
2022-12-01T11:13:32.5877539Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105928.xml
2022-12-01T11:13:32.5878502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5878952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5879975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5880461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5881116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxe8j2i7h
2022-12-01T11:13:32.5881782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxe8j2i7h/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5881827Z 
2022-12-01T11:13:32.5882079Z Running tests...
2022-12-01T11:13:32.5882724Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5883673Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5884441Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5885121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44955
2022-12-01T11:13:32.5885690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44956
2022-12-01T11:13:32.5886329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44957
2022-12-01T11:13:32.5886861Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44958
2022-12-01T11:13:32.5887853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5888291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5889290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5889760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5890719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5891176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5892166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5892645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5893583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5894032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5895556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5896035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5897005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5897447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5898466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5898937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5899575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq_xbydn
2022-12-01T11:13:32.5900267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq_xbydn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5900930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsolxcy_c
2022-12-01T11:13:32.5901625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsolxcy_c/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5902286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7mxg0c_
2022-12-01T11:13:32.5902966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7mxg0c_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5903640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa72zjw4m
2022-12-01T11:13:32.5904350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa72zjw4m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5904928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5905496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5906089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5906723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5907075Z fi_getinfo: -61
2022-12-01T11:13:32.5907407Z fi_getinfo: -61
2022-12-01T11:13:32.5907739Z fi_getinfo: -61
2022-12-01T11:13:32.5908078Z fi_getinfo: -61
2022-12-01T11:13:32.5908421Z ok (12.797s)
2022-12-01T11:13:32.5908478Z 
2022-12-01T11:13:32.5909134Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5909414Z Ran 1 test in 12.797s
2022-12-01T11:13:32.5909572Z 
2022-12-01T11:13:32.5909781Z OK
2022-12-01T11:13:32.5909814Z 
2022-12-01T11:13:32.5910110Z Generating XML reports...
2022-12-01T11:13:32.5911566Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105950.xml
2022-12-01T11:13:32.5912524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5912966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5913960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5914424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5915083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mhov12b
2022-12-01T11:13:32.5915777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mhov12b/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5915823Z 
2022-12-01T11:13:32.5916070Z Running tests...
2022-12-01T11:13:32.5916707Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5917652Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5918446Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5918989Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45314
2022-12-01T11:13:32.5919534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45315
2022-12-01T11:13:32.5920096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45316
2022-12-01T11:13:32.5920658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45317
2022-12-01T11:13:32.5921660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5922079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5923081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5923562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5924539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5924979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5925935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5926412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5927379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5927838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5928825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5929280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5930254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5930687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5931681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5932255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5932922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9sm1knz2
2022-12-01T11:13:32.5933698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9sm1knz2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5934307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6x7z24y9
2022-12-01T11:13:32.5935577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6x7z24y9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5936239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5rhxg72
2022-12-01T11:13:32.5936970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5rhxg72/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5937633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphu0j6mcu
2022-12-01T11:13:32.5938288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphu0j6mcu/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5938872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5939464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5940047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5940632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5941002Z fi_getinfo: -61
2022-12-01T11:13:32.5941325Z fi_getinfo: -61
2022-12-01T11:13:32.5941661Z fi_getinfo: -61
2022-12-01T11:13:32.5941980Z fi_getinfo: -61
2022-12-01T11:13:32.5942221Z ok (11.260s)
2022-12-01T11:13:32.5942256Z 
2022-12-01T11:13:32.5942912Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5943186Z Ran 1 test in 11.260s
2022-12-01T11:13:32.5943216Z 
2022-12-01T11:13:32.5943425Z OK
2022-12-01T11:13:32.5943472Z 
2022-12-01T11:13:32.5943759Z Generating XML reports...
2022-12-01T11:13:32.5945227Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110005.xml
2022-12-01T11:13:32.5946218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5946639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5947639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5948114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5948794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoioyryt0
2022-12-01T11:13:32.5949490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoioyryt0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5949535Z 
2022-12-01T11:13:32.5949811Z Running tests...
2022-12-01T11:13:32.5950458Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5951424Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5952167Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5952715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45668
2022-12-01T11:13:32.5953269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45669
2022-12-01T11:13:32.5953829Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45670
2022-12-01T11:13:32.5954381Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45671
2022-12-01T11:13:32.5955348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5955966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5956974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5957564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5958519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5958956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5959931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5960401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5961351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5961774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5962768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5963239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5964191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5964624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5965624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5966089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5966769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpku77jq34
2022-12-01T11:13:32.5967421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xgg30l7
2022-12-01T11:13:32.5968097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpku77jq34/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5968784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xgg30l7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5969452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbriocpef
2022-12-01T11:13:32.5970119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbriocpef/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5970775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5dhdh4d
2022-12-01T11:13:32.5971481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5dhdh4d/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5972071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.5972649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.5973241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.5973840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.5974180Z fi_getinfo: -61
2022-12-01T11:13:32.5975092Z fi_getinfo: -61
2022-12-01T11:13:32.5975449Z fi_getinfo: -61
2022-12-01T11:13:32.5975773Z fi_getinfo: -61
2022-12-01T11:13:32.5976006Z ok (4.394s)
2022-12-01T11:13:32.5976051Z 
2022-12-01T11:13:32.5976727Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5976969Z Ran 1 test in 4.395s
2022-12-01T11:13:32.5977027Z 
2022-12-01T11:13:32.5977214Z OK
2022-12-01T11:13:32.5977259Z 
2022-12-01T11:13:32.5977556Z Generating XML reports...
2022-12-01T11:13:32.5978979Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110019.xml
2022-12-01T11:13:32.5980024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5980612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5981621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5982250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5982902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8oid34oo
2022-12-01T11:13:32.5983576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8oid34oo/_remote_module_non_scriptable.py
2022-12-01T11:13:32.5983647Z 
2022-12-01T11:13:32.5983879Z Running tests...
2022-12-01T11:13:32.5984546Z ----------------------------------------------------------------------
2022-12-01T11:13:32.5985500Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.5986303Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.5986852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46011
2022-12-01T11:13:32.5987422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46012
2022-12-01T11:13:32.5987989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46013
2022-12-01T11:13:32.5988540Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46014
2022-12-01T11:13:32.5989497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5989947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5990948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5991418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5992396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5992814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5993805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5994286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5995206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5995627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5996616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5997079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.5998057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.5998497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.5999495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.5999962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6000641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jqgstdo
2022-12-01T11:13:32.6001322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jqgstdo/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6001987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6sz4b6vt
2022-12-01T11:13:32.6002666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6sz4b6vt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6003328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8xi0nu3
2022-12-01T11:13:32.6004113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8xi0nu3/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6004777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkjhhs2r1
2022-12-01T11:13:32.6005567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkjhhs2r1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6006146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6006760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6007352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6007934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6008264Z fi_getinfo: -61
2022-12-01T11:13:32.6008596Z fi_getinfo: -61
2022-12-01T11:13:32.6008925Z fi_getinfo: -61
2022-12-01T11:13:32.6009251Z fi_getinfo: -61
2022-12-01T11:13:32.6009477Z ok (7.189s)
2022-12-01T11:13:32.6009541Z 
2022-12-01T11:13:32.6010174Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6010440Z Ran 1 test in 7.190s
2022-12-01T11:13:32.6010474Z 
2022-12-01T11:13:32.6010710Z OK
2022-12-01T11:13:32.6010739Z 
2022-12-01T11:13:32.6011039Z Generating XML reports...
2022-12-01T11:13:32.6012502Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110025.xml
2022-12-01T11:13:32.6013508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6013950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6015683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6016152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6016828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal1251wp
2022-12-01T11:13:32.6017503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal1251wp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6017556Z 
2022-12-01T11:13:32.6017811Z Running tests...
2022-12-01T11:13:32.6018476Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6019427Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6020243Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6020777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46362
2022-12-01T11:13:32.6021344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46363
2022-12-01T11:13:32.6021891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46364
2022-12-01T11:13:32.6022450Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46365
2022-12-01T11:13:32.6023417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6023868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6024864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6025360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6026330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6026757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6027724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6028359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6029334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6029888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6030872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6031357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6032302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6032728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6033708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6034170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6034805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48mcmlk1
2022-12-01T11:13:32.6035533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48mcmlk1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6036193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufp9uxfi
2022-12-01T11:13:32.6036887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufp9uxfi/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6037550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsamd3ny6
2022-12-01T11:13:32.6038231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsamd3ny6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6038882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rv_ddnl
2022-12-01T11:13:32.6039544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rv_ddnl/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6040143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6040708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6041309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6041656Z fi_getinfo: -61
2022-12-01T11:13:32.6041985Z fi_getinfo: -61
2022-12-01T11:13:32.6042556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6042882Z fi_getinfo: -61
2022-12-01T11:13:32.6043183Z fi_getinfo: -61
2022-12-01T11:13:32.6043417Z ok (7.450s)
2022-12-01T11:13:32.6043450Z 
2022-12-01T11:13:32.6044134Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6044397Z Ran 1 test in 7.450s
2022-12-01T11:13:32.6044431Z 
2022-12-01T11:13:32.6044643Z OK
2022-12-01T11:13:32.6044680Z 
2022-12-01T11:13:32.6044996Z Generating XML reports...
2022-12-01T11:13:32.6046441Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110035.xml
2022-12-01T11:13:32.6047432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6047847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6048849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6049316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6049967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomgpx8ub
2022-12-01T11:13:32.6050675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomgpx8ub/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6050714Z 
2022-12-01T11:13:32.6051067Z Running tests...
2022-12-01T11:13:32.6051732Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6052692Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6053548Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6054078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46713
2022-12-01T11:13:32.6055209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46714
2022-12-01T11:13:32.6055788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46715
2022-12-01T11:13:32.6056324Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46716
2022-12-01T11:13:32.6057325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6057762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6058720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6059144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6060137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6060610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6061605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6062072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6063064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6063497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6064496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6064990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6065927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6066345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6067333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6067805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6068451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd2o80tv0
2022-12-01T11:13:32.6069129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod0n2t33
2022-12-01T11:13:32.6069836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd2o80tv0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6070512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod0n2t33/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6071176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpay67wctj
2022-12-01T11:13:32.6071838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpay67wctj/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6072485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnor75aye
2022-12-01T11:13:32.6073159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnor75aye/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6073762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6074335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6075050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6075674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6076222Z fi_getinfo: -61
2022-12-01T11:13:32.6076525Z fi_getinfo: -61
2022-12-01T11:13:32.6076863Z fi_getinfo: -61
2022-12-01T11:13:32.6077214Z fi_getinfo: -61
2022-12-01T11:13:32.6077429Z ok (7.298s)
2022-12-01T11:13:32.6077477Z 
2022-12-01T11:13:32.6078116Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6078374Z Ran 1 test in 7.298s
2022-12-01T11:13:32.6078404Z 
2022-12-01T11:13:32.6078619Z OK
2022-12-01T11:13:32.6078664Z 
2022-12-01T11:13:32.6078960Z Generating XML reports...
2022-12-01T11:13:32.6080402Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110044.xml
2022-12-01T11:13:32.6081386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6081818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6082828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6083308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6083973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7l1ya8z
2022-12-01T11:13:32.6084668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7l1ya8z/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6084718Z 
2022-12-01T11:13:32.6084959Z Running tests...
2022-12-01T11:13:32.6085611Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6086518Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6087372Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6089396Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.585s)
2022-12-01T11:13:32.6089444Z 
2022-12-01T11:13:32.6090091Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6090345Z Ran 1 test in 1.585s
2022-12-01T11:13:32.6090390Z 
2022-12-01T11:13:32.6090663Z OK (skipped=1)
2022-12-01T11:13:32.6090693Z 
2022-12-01T11:13:32.6090988Z Generating XML reports...
2022-12-01T11:13:32.6092416Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110054.xml
2022-12-01T11:13:32.6093404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6093837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6095233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6095739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6096389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdsgf_ej
2022-12-01T11:13:32.6097086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdsgf_ej/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6097130Z 
2022-12-01T11:13:32.6097401Z Running tests...
2022-12-01T11:13:32.6098050Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6098983Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6099893Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6100447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47094
2022-12-01T11:13:32.6101130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47095
2022-12-01T11:13:32.6101670Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47096
2022-12-01T11:13:32.6102219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47097
2022-12-01T11:13:32.6103217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6103657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6104630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6105103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6106087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6106543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6107562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6108037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6109009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6109443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6110426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6110905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6111876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6112278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6113299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6113769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6114435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9orm7rv
2022-12-01T11:13:32.6115150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9orm7rv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6115789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv78yfmr
2022-12-01T11:13:32.6116480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv78yfmr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6117138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkj2f2xq
2022-12-01T11:13:32.6117820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkj2f2xq/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6118444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv9ih613
2022-12-01T11:13:32.6119124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv9ih613/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6119735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6120336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6120925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6121518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6121851Z fi_getinfo: -61
2022-12-01T11:13:32.6122191Z fi_getinfo: -61
2022-12-01T11:13:32.6122497Z fi_getinfo: -61
2022-12-01T11:13:32.6122909Z fi_getinfo: -61
2022-12-01T11:13:32.6123150Z ok (9.880s)
2022-12-01T11:13:32.6123195Z 
2022-12-01T11:13:32.6123846Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6124206Z Ran 1 test in 9.881s
2022-12-01T11:13:32.6124235Z 
2022-12-01T11:13:32.6124458Z OK
2022-12-01T11:13:32.6124501Z 
2022-12-01T11:13:32.6124789Z Generating XML reports...
2022-12-01T11:13:32.6126233Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110058.xml
2022-12-01T11:13:32.6127173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6127618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6128602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6129064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6129730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_6lfthn
2022-12-01T11:13:32.6130415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_6lfthn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6130458Z 
2022-12-01T11:13:32.6130717Z Running tests...
2022-12-01T11:13:32.6131390Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6132302Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6133070Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6133625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47445
2022-12-01T11:13:32.6134180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47446
2022-12-01T11:13:32.6135229Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47447
2022-12-01T11:13:32.6135812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47448
2022-12-01T11:13:32.6136797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6137253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6138246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6138688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6139651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6140065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6141078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6141542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6142501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6142950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6143941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6144393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6145379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6145828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6146812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6147446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6148115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklj4lu0m
2022-12-01T11:13:32.6148944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklj4lu0m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6149617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2pksy1f1
2022-12-01T11:13:32.6150308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2pksy1f1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6150952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdltlklv
2022-12-01T11:13:32.6151648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdltlklv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6152286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1bt21n5a
2022-12-01T11:13:32.6152978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1bt21n5a/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6153564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6154164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6154736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6155322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6155660Z fi_getinfo: -61
2022-12-01T11:13:32.6155979Z fi_getinfo: -61
2022-12-01T11:13:32.6156313Z fi_getinfo: -61
2022-12-01T11:13:32.6156626Z fi_getinfo: -61
2022-12-01T11:13:32.6156860Z ok (9.699s)
2022-12-01T11:13:32.6156895Z 
2022-12-01T11:13:32.6157557Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6157834Z Ran 1 test in 9.699s
2022-12-01T11:13:32.6157864Z 
2022-12-01T11:13:32.6158049Z OK
2022-12-01T11:13:32.6158112Z 
2022-12-01T11:13:32.6158387Z Generating XML reports...
2022-12-01T11:13:32.6159798Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110110.xml
2022-12-01T11:13:32.6160787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6161216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6162222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6162702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6163373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjos5l_w9
2022-12-01T11:13:32.6164073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjos5l_w9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6164116Z 
2022-12-01T11:13:32.6164346Z Running tests...
2022-12-01T11:13:32.6165017Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6165958Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6166747Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6167300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47796
2022-12-01T11:13:32.6167860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47797
2022-12-01T11:13:32.6168403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47798
2022-12-01T11:13:32.6168943Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47799
2022-12-01T11:13:32.6169928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6170458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6171475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6172054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6173017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6173443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6174959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6175460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6176421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6176829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6177801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6178293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6179268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6179688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6180657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6181133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6181810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav0uq3m2
2022-12-01T11:13:32.6182501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav0uq3m2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6183143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7oa7qrdr
2022-12-01T11:13:32.6183807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7oa7qrdr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6184466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6p2kyyi
2022-12-01T11:13:32.6185143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6p2kyyi/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6185792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5w0xwglb
2022-12-01T11:13:32.6186472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5w0xwglb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6187058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6187640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6188220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6188811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6189169Z fi_getinfo: -61
2022-12-01T11:13:32.6189501Z fi_getinfo: -61
2022-12-01T11:13:32.6189826Z fi_getinfo: -61
2022-12-01T11:13:32.6190163Z fi_getinfo: -61
2022-12-01T11:13:32.6190385Z ok (9.701s)
2022-12-01T11:13:32.6190420Z 
2022-12-01T11:13:32.6191058Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6191323Z Ran 1 test in 9.701s
2022-12-01T11:13:32.6191363Z 
2022-12-01T11:13:32.6191576Z OK
2022-12-01T11:13:32.6191612Z 
2022-12-01T11:13:32.6191907Z Generating XML reports...
2022-12-01T11:13:32.6193385Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110122.xml
2022-12-01T11:13:32.6194498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6194954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6196070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6196553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6197204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad2ra6jw
2022-12-01T11:13:32.6197901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad2ra6jw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6197946Z 
2022-12-01T11:13:32.6198184Z Running tests...
2022-12-01T11:13:32.6198835Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6199764Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6200514Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6201088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48147
2022-12-01T11:13:32.6201651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48148
2022-12-01T11:13:32.6202186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48149
2022-12-01T11:13:32.6202706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48150
2022-12-01T11:13:32.6203668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6204100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6205091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6205577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6206592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6207061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6208045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6208493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6209486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6209934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6210917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6211401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6212365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6212807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6213812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6214285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6215605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9l8y7ue8
2022-12-01T11:13:32.6216292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9l8y7ue8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6216964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8w9cp2n
2022-12-01T11:13:32.6217657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8w9cp2n/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6218458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjo8_y31
2022-12-01T11:13:32.6219155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjo8_y31/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6219947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcggn0439
2022-12-01T11:13:32.6220624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcggn0439/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6221199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6221802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6222382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6222972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6223334Z fi_getinfo: -61
2022-12-01T11:13:32.6223660Z fi_getinfo: -61
2022-12-01T11:13:32.6224008Z fi_getinfo: -61
2022-12-01T11:13:32.6224324Z fi_getinfo: -61
2022-12-01T11:13:32.6224537Z ok (9.715s)
2022-12-01T11:13:32.6224573Z 
2022-12-01T11:13:32.6225224Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6225499Z Ran 1 test in 9.715s
2022-12-01T11:13:32.6225527Z 
2022-12-01T11:13:32.6225742Z OK
2022-12-01T11:13:32.6225775Z 
2022-12-01T11:13:32.6226086Z Generating XML reports...
2022-12-01T11:13:32.6227543Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110134.xml
2022-12-01T11:13:32.6228519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6228952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6229930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6230432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6231080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zc_0swd
2022-12-01T11:13:32.6231770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zc_0swd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6231805Z 
2022-12-01T11:13:32.6232060Z Running tests...
2022-12-01T11:13:32.6232705Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6233649Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6234452Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6235035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48498
2022-12-01T11:13:32.6235560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48499
2022-12-01T11:13:32.6236124Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48500
2022-12-01T11:13:32.6236669Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48501
2022-12-01T11:13:32.6237640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6238082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6239083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6239570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6240525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6240920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6242016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6242502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6243542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6243964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6244946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6245443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6246418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6246872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6247834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6248320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6248983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_hg4j2l
2022-12-01T11:13:32.6249681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_hg4j2l/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6250330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyevf3hat
2022-12-01T11:13:32.6251022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyevf3hat/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6251686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphodyljhb
2022-12-01T11:13:32.6252366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphodyljhb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6253014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlvnkizo
2022-12-01T11:13:32.6253721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlvnkizo/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6254305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6255528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6256132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6256726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6257083Z fi_getinfo: -61
2022-12-01T11:13:32.6257381Z fi_getinfo: -61
2022-12-01T11:13:32.6257682Z fi_getinfo: -61
2022-12-01T11:13:32.6258017Z fi_getinfo: -61
2022-12-01T11:13:32.6258248Z ok (9.801s)
2022-12-01T11:13:32.6258289Z 
2022-12-01T11:13:32.6258926Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6259200Z Ran 1 test in 9.801s
2022-12-01T11:13:32.6259232Z 
2022-12-01T11:13:32.6259436Z OK
2022-12-01T11:13:32.6259480Z 
2022-12-01T11:13:32.6259787Z Generating XML reports...
2022-12-01T11:13:32.6261238Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110145.xml
2022-12-01T11:13:32.6262178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6262604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6263623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6264118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6264762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppuwviqh7
2022-12-01T11:13:32.6265477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppuwviqh7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6265675Z 
2022-12-01T11:13:32.6265959Z Running tests...
2022-12-01T11:13:32.6266611Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6267584Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6268366Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6268906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48857
2022-12-01T11:13:32.6269467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48858
2022-12-01T11:13:32.6270010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 48859
2022-12-01T11:13:32.6270553Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 48860
2022-12-01T11:13:32.6271543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6271995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6272979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6273440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6274393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6274820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6275811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6276304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6277229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6277678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6278661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6279119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6280136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6280593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6281577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6282074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6282737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03830uzi
2022-12-01T11:13:32.6283437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03830uzi/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6284090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylqkpa4a
2022-12-01T11:13:32.6284800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylqkpa4a/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6285422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohrq6f6m
2022-12-01T11:13:32.6286116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohrq6f6m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6286764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph00tzwa0
2022-12-01T11:13:32.6287453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph00tzwa0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6288042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6288645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6289352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6289963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6290376Z fi_getinfo: -61
2022-12-01T11:13:32.6290717Z fi_getinfo: -61
2022-12-01T11:13:32.6291050Z fi_getinfo: -61
2022-12-01T11:13:32.6291373Z fi_getinfo: -61
2022-12-01T11:13:32.6291609Z ok (9.709s)
2022-12-01T11:13:32.6291653Z 
2022-12-01T11:13:32.6292305Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6292565Z Ran 1 test in 9.710s
2022-12-01T11:13:32.6292607Z 
2022-12-01T11:13:32.6292800Z OK
2022-12-01T11:13:32.6292844Z 
2022-12-01T11:13:32.6293109Z Generating XML reports...
2022-12-01T11:13:32.6294853Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110157.xml
2022-12-01T11:13:32.6295872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6296318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6297331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6297813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6298468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvjxg43q2
2022-12-01T11:13:32.6299180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvjxg43q2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6299215Z 
2022-12-01T11:13:32.6299474Z Running tests...
2022-12-01T11:13:32.6300106Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6301063Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6301867Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6302423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49216
2022-12-01T11:13:32.6302994Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49217
2022-12-01T11:13:32.6303544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49218
2022-12-01T11:13:32.6304100Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49219
2022-12-01T11:13:32.6305069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6305487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6306532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6307031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6307998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6308427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6309424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6309916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6310862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6311272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6312262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6312733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6313872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6314323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6315466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6315947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6316626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7t6pv8oz
2022-12-01T11:13:32.6317327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7t6pv8oz/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6317964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66p8v_u6
2022-12-01T11:13:32.6318654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66p8v_u6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6319313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyji56rid
2022-12-01T11:13:32.6320010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyji56rid/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6320677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0tb_ucd
2022-12-01T11:13:32.6321362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0tb_ucd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6321966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6322560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6323164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6323723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6324084Z fi_getinfo: -61
2022-12-01T11:13:32.6324419Z fi_getinfo: -61
2022-12-01T11:13:32.6324755Z fi_getinfo: -61
2022-12-01T11:13:32.6325101Z fi_getinfo: -61
2022-12-01T11:13:32.6325324Z ok (9.603s)
2022-12-01T11:13:32.6325371Z 
2022-12-01T11:13:32.6326037Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6326276Z Ran 1 test in 9.603s
2022-12-01T11:13:32.6326313Z 
2022-12-01T11:13:32.6326537Z OK
2022-12-01T11:13:32.6326571Z 
2022-12-01T11:13:32.6326867Z Generating XML reports...
2022-12-01T11:13:32.6328327Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110209.xml
2022-12-01T11:13:32.6329329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6329767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6330769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6331258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6331888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fzmuqkm
2022-12-01T11:13:32.6332601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fzmuqkm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6332632Z 
2022-12-01T11:13:32.6332902Z Running tests...
2022-12-01T11:13:32.6333572Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6335183Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6335983Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6336542Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49575
2022-12-01T11:13:32.6337097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49576
2022-12-01T11:13:32.6337798Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49577
2022-12-01T11:13:32.6338344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49578
2022-12-01T11:13:32.6339440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6339877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6340877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6341368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6342324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6342760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6343766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6344223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6345175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6345614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6346610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6347088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6348042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6348499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6349494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6349978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6350619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sv5tbql
2022-12-01T11:13:32.6351319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sv5tbql/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6351965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbx_74cmb
2022-12-01T11:13:32.6352648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbx_74cmb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6353302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xp7w9o7
2022-12-01T11:13:32.6354006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xp7w9o7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6354665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsizumrb
2022-12-01T11:13:32.6355361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsizumrb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6355928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6356530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6357128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6357710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6358056Z fi_getinfo: -61
2022-12-01T11:13:32.6358400Z fi_getinfo: -61
2022-12-01T11:13:32.6358716Z fi_getinfo: -61
2022-12-01T11:13:32.6359040Z fi_getinfo: -61
2022-12-01T11:13:32.6359241Z ok (9.763s)
2022-12-01T11:13:32.6359290Z 
2022-12-01T11:13:32.6359944Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6360220Z Ran 1 test in 9.763s
2022-12-01T11:13:32.6360256Z 
2022-12-01T11:13:32.6360570Z OK
2022-12-01T11:13:32.6360616Z 
2022-12-01T11:13:32.6360933Z Generating XML reports...
2022-12-01T11:13:32.6362383Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110221.xml
2022-12-01T11:13:32.6363465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6363914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6364896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6365394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6366033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk00eoat1
2022-12-01T11:13:32.6366742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk00eoat1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6366785Z 
2022-12-01T11:13:32.6367053Z Running tests...
2022-12-01T11:13:32.6367720Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6368663Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6369473Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6370046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49934
2022-12-01T11:13:32.6370590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49935
2022-12-01T11:13:32.6371145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49936
2022-12-01T11:13:32.6371698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49937
2022-12-01T11:13:32.6372684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6373136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6374129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6375270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6376257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6376670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6377676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6378157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6379082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6379525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6380516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6380982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6381951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6382377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6383328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6383801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6384451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr539cuav
2022-12-01T11:13:32.6385293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr539cuav/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6385944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlnwingb
2022-12-01T11:13:32.6386774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlnwingb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6387452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fm9922w
2022-12-01T11:13:32.6388110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fm9922w/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6388743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75woh2pm
2022-12-01T11:13:32.6389447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75woh2pm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6390033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6390640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6391230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6391814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6392184Z fi_getinfo: -61
2022-12-01T11:13:32.6392515Z fi_getinfo: -61
2022-12-01T11:13:32.6392813Z fi_getinfo: -61
2022-12-01T11:13:32.6393153Z fi_getinfo: -61
2022-12-01T11:13:32.6393380Z ok (9.806s)
2022-12-01T11:13:32.6393418Z 
2022-12-01T11:13:32.6394085Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6394356Z Ran 1 test in 9.806s
2022-12-01T11:13:32.6394401Z 
2022-12-01T11:13:32.6394607Z OK
2022-12-01T11:13:32.6394645Z 
2022-12-01T11:13:32.6394956Z Generating XML reports...
2022-12-01T11:13:32.6396416Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110233.xml
2022-12-01T11:13:32.6397382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6397818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6398804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6399297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6399962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u2thxz0
2022-12-01T11:13:32.6400657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u2thxz0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6400702Z 
2022-12-01T11:13:32.6400948Z Running tests...
2022-12-01T11:13:32.6401615Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6402550Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6403372Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6403900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50285
2022-12-01T11:13:32.6404461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50286
2022-12-01T11:13:32.6405003Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50287
2022-12-01T11:13:32.6405560Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50288
2022-12-01T11:13:32.6406582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6407038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6408061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6408609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6409591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6410111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6411103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6411596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6412554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6412984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6413997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6414956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6415936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6416378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6417369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6417839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6418498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_51e8w_w
2022-12-01T11:13:32.6419187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_51e8w_w/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6419836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3c7dmoy
2022-12-01T11:13:32.6420528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3c7dmoy/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6421157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp4z4mv9
2022-12-01T11:13:32.6421860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp4z4mv9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6422531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sp9hjdg
2022-12-01T11:13:32.6423222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sp9hjdg/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6423827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6424431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6425022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6425602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6425963Z fi_getinfo: -61
2022-12-01T11:13:32.6426272Z fi_getinfo: -61
2022-12-01T11:13:32.6426593Z fi_getinfo: -61
2022-12-01T11:13:32.6426926Z fi_getinfo: -61
2022-12-01T11:13:32.6427164Z ok (9.676s)
2022-12-01T11:13:32.6427201Z 
2022-12-01T11:13:32.6427855Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6428118Z Ran 1 test in 9.676s
2022-12-01T11:13:32.6428166Z 
2022-12-01T11:13:32.6428408Z OK
2022-12-01T11:13:32.6428445Z 
2022-12-01T11:13:32.6428728Z Generating XML reports...
2022-12-01T11:13:32.6430173Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110245.xml
2022-12-01T11:13:32.6431144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6431586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6432732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6433230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6433892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqjkd8g7
2022-12-01T11:13:32.6434713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqjkd8g7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6434744Z 
2022-12-01T11:13:32.6435013Z Running tests...
2022-12-01T11:13:32.6435654Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6436602Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6437412Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6437979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50636
2022-12-01T11:13:32.6438549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50637
2022-12-01T11:13:32.6439106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50638
2022-12-01T11:13:32.6439669Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50639
2022-12-01T11:13:32.6440668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6441095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6442125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6442612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6443590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6444019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6445024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6445508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6446489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6446937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6447908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6448383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6449349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6449789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6450798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6451284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6451968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfcakt4i
2022-12-01T11:13:32.6452687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfcakt4i/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6453312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv805r6gk
2022-12-01T11:13:32.6453992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv805r6gk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6455181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8hkdnzz
2022-12-01T11:13:32.6455897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8hkdnzz/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6456566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzlab7lki
2022-12-01T11:13:32.6457486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzlab7lki/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6458094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6458816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6459424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6459982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6460357Z fi_getinfo: -61
2022-12-01T11:13:32.6460692Z fi_getinfo: -61
2022-12-01T11:13:32.6461025Z fi_getinfo: -61
2022-12-01T11:13:32.6461363Z fi_getinfo: -61
2022-12-01T11:13:32.6461595Z ok (9.713s)
2022-12-01T11:13:32.6461635Z 
2022-12-01T11:13:32.6462285Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6462526Z Ran 1 test in 9.713s
2022-12-01T11:13:32.6462564Z 
2022-12-01T11:13:32.6462801Z OK
2022-12-01T11:13:32.6462833Z 
2022-12-01T11:13:32.6463148Z Generating XML reports...
2022-12-01T11:13:32.6464584Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110257.xml
2022-12-01T11:13:32.6465578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6466014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6467030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6467522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6468187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnzm7m2f
2022-12-01T11:13:32.6468857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnzm7m2f/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6468902Z 
2022-12-01T11:13:32.6469158Z Running tests...
2022-12-01T11:13:32.6469827Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6470763Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6471541Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6472130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50987
2022-12-01T11:13:32.6472696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50988
2022-12-01T11:13:32.6473248Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50989
2022-12-01T11:13:32.6473768Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50990
2022-12-01T11:13:32.6474751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6475192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6476202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6476694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6477659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6478089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6479091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6479582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6480626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6481076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6482078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6482686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6483681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6484109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6485098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6485600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6486222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe729nj82
2022-12-01T11:13:32.6486923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe729nj82/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6487591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uh34bbp
2022-12-01T11:13:32.6488289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uh34bbp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6488945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v8j4cdt
2022-12-01T11:13:32.6489618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v8j4cdt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6490270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9a26l74l
2022-12-01T11:13:32.6490957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9a26l74l/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6491557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6492138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6492738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6493335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6493693Z fi_getinfo: -61
2022-12-01T11:13:32.6494018Z fi_getinfo: -61
2022-12-01T11:13:32.6494307Z fi_getinfo: -61
2022-12-01T11:13:32.6495234Z fi_getinfo: -61
2022-12-01T11:13:32.6495444Z ok (9.702s)
2022-12-01T11:13:32.6495473Z 
2022-12-01T11:13:32.6496146Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6496418Z Ran 1 test in 9.703s
2022-12-01T11:13:32.6496455Z 
2022-12-01T11:13:32.6496669Z OK
2022-12-01T11:13:32.6496702Z 
2022-12-01T11:13:32.6497005Z Generating XML reports...
2022-12-01T11:13:32.6498451Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110309.xml
2022-12-01T11:13:32.6499436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6499883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6500889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6501335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6502006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ula55c2
2022-12-01T11:13:32.6502711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ula55c2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6502743Z 
2022-12-01T11:13:32.6502994Z Running tests...
2022-12-01T11:13:32.6503675Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6504761Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6505583Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6506282Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51338
2022-12-01T11:13:32.6506847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51339
2022-12-01T11:13:32.6507417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51340
2022-12-01T11:13:32.6507972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51341
2022-12-01T11:13:32.6508957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6509392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6510390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6510894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6511872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6512318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6513292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6513783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6514736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6515173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6516150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6516623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6517595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6518045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6518983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6519449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6520060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9mqs0k9
2022-12-01T11:13:32.6520764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9mqs0k9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6521436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6ius71g
2022-12-01T11:13:32.6522121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6ius71g/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6522797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb1m5p2o7
2022-12-01T11:13:32.6523492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb1m5p2o7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6524132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_shkz4qd
2022-12-01T11:13:32.6524806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_shkz4qd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6525401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6526002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6526599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6527193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6527552Z fi_getinfo: -61
2022-12-01T11:13:32.6527999Z fi_getinfo: -61
2022-12-01T11:13:32.6528359Z fi_getinfo: -61
2022-12-01T11:13:32.6528678Z fi_getinfo: -61
2022-12-01T11:13:32.6528916Z ok (9.787s)
2022-12-01T11:13:32.6529038Z 
2022-12-01T11:13:32.6529707Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6529972Z Ran 1 test in 9.787s
2022-12-01T11:13:32.6530017Z 
2022-12-01T11:13:32.6530241Z OK
2022-12-01T11:13:32.6530275Z 
2022-12-01T11:13:32.6530567Z Generating XML reports...
2022-12-01T11:13:32.6532014Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110321.xml
2022-12-01T11:13:32.6533017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6533431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6534841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6535339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6536023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyasix9t
2022-12-01T11:13:32.6536737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyasix9t/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6536782Z 
2022-12-01T11:13:32.6537039Z Running tests...
2022-12-01T11:13:32.6537688Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6538649Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6539416Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6540005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51689
2022-12-01T11:13:32.6540565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51690
2022-12-01T11:13:32.6541118Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51691
2022-12-01T11:13:32.6541705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51692
2022-12-01T11:13:32.6542705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6543128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6544133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6544629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6545590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6546040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6547052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6547548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6548524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6548962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6549953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6550436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6551395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6551836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6552965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6553463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6554229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjfoyx2l
2022-12-01T11:13:32.6554949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjfoyx2l/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6555637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd98r8qm2
2022-12-01T11:13:32.6556341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd98r8qm2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6557048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkkxyr8r
2022-12-01T11:13:32.6557690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkkxyr8r/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6558368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelkkbha8
2022-12-01T11:13:32.6559070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelkkbha8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6559678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6560262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6560866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6561473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6561828Z fi_getinfo: -61
2022-12-01T11:13:32.6562147Z fi_getinfo: -61
2022-12-01T11:13:32.6562483Z fi_getinfo: -61
2022-12-01T11:13:32.6562816Z fi_getinfo: -61
2022-12-01T11:13:32.6563052Z ok (9.703s)
2022-12-01T11:13:32.6563081Z 
2022-12-01T11:13:32.6563752Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6564035Z Ran 1 test in 9.703s
2022-12-01T11:13:32.6564068Z 
2022-12-01T11:13:32.6564294Z OK
2022-12-01T11:13:32.6564335Z 
2022-12-01T11:13:32.6564662Z Generating XML reports...
2022-12-01T11:13:32.6566077Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110333.xml
2022-12-01T11:13:32.6567061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6567502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6568555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6569011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6569656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkcto3ml
2022-12-01T11:13:32.6570357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkcto3ml/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6570407Z 
2022-12-01T11:13:32.6570653Z Running tests...
2022-12-01T11:13:32.6571302Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6572269Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6573062Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6573638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52040
2022-12-01T11:13:32.6574200Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52041
2022-12-01T11:13:32.6575407Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52042
2022-12-01T11:13:32.6575990Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52043
2022-12-01T11:13:32.6577131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6577587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6578711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6579193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6580257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6580698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6581694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6582207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6583154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6583590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6584546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6585039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6586006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6586438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6587434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6587900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6588560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6quut2s
2022-12-01T11:13:32.6589289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6quut2s/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6589949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv80x9qt7
2022-12-01T11:13:32.6590597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv80x9qt7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6591279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2r9agv0
2022-12-01T11:13:32.6591962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2r9agv0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6592619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcaf7x8s
2022-12-01T11:13:32.6593280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcaf7x8s/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6593888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6594498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6595091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6595660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6596033Z fi_getinfo: -61
2022-12-01T11:13:32.6596362Z fi_getinfo: -61
2022-12-01T11:13:32.6596695Z fi_getinfo: -61
2022-12-01T11:13:32.6597022Z fi_getinfo: -61
2022-12-01T11:13:32.6597263Z ok (9.693s)
2022-12-01T11:13:32.6597295Z 
2022-12-01T11:13:32.6597956Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6598190Z Ran 1 test in 9.693s
2022-12-01T11:13:32.6598265Z 
2022-12-01T11:13:32.6598447Z OK
2022-12-01T11:13:32.6598473Z 
2022-12-01T11:13:32.6598774Z Generating XML reports...
2022-12-01T11:13:32.6600323Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110345.xml
2022-12-01T11:13:32.6601306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6601842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6602865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6603339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6603993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hl4baxl
2022-12-01T11:13:32.6604662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hl4baxl/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6604722Z 
2022-12-01T11:13:32.6604959Z Running tests...
2022-12-01T11:13:32.6605642Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6606619Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6607433Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6608008Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52391
2022-12-01T11:13:32.6608565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52392
2022-12-01T11:13:32.6609134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52393
2022-12-01T11:13:32.6609709Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52394
2022-12-01T11:13:32.6610667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6611115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6612124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6612609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6613582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6614021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6615650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6616151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6617090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6617526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6618504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6618997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6619970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6620413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6621397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6621881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6622555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcr8jhm81
2022-12-01T11:13:32.6623222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcr8jhm81/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6623879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjx34k2e
2022-12-01T11:13:32.6624727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjx34k2e/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6625408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2czc9cc
2022-12-01T11:13:32.6626213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2czc9cc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6626855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprozay_k2
2022-12-01T11:13:32.6627546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprozay_k2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6628136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6628717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6629313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6629895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6630241Z fi_getinfo: -61
2022-12-01T11:13:32.6630597Z fi_getinfo: -61
2022-12-01T11:13:32.6630926Z fi_getinfo: -61
2022-12-01T11:13:32.6631265Z fi_getinfo: -61
2022-12-01T11:13:32.6631476Z ok (9.715s)
2022-12-01T11:13:32.6631542Z 
2022-12-01T11:13:32.6632181Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6632455Z Ran 1 test in 9.715s
2022-12-01T11:13:32.6632486Z 
2022-12-01T11:13:32.6632703Z OK
2022-12-01T11:13:32.6632741Z 
2022-12-01T11:13:32.6633049Z Generating XML reports...
2022-12-01T11:13:32.6634507Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110357.xml
2022-12-01T11:13:32.6635474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6635925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6636943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6637408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6638069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9iae4grc
2022-12-01T11:13:32.6638767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9iae4grc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6638814Z 
2022-12-01T11:13:32.6639075Z Running tests...
2022-12-01T11:13:32.6639760Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6640709Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6641500Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6642072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52742
2022-12-01T11:13:32.6642654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52743
2022-12-01T11:13:32.6643199Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52744
2022-12-01T11:13:32.6643777Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52745
2022-12-01T11:13:32.6644780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6645208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6646191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6646634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6647638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6648123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6649196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6649821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6650772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6651229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6652210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6652707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6653659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6654084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6655815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6656265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6656948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpalfxr3e2
2022-12-01T11:13:32.6657639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpalfxr3e2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6658304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpefb87_1h
2022-12-01T11:13:32.6658951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4x4p75e
2022-12-01T11:13:32.6659641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpefb87_1h/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6660339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4x4p75e/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6660974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ijl2sp5
2022-12-01T11:13:32.6661637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ijl2sp5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6662250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6662834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6663426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6664037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6664385Z fi_getinfo: -61
2022-12-01T11:13:32.6664711Z fi_getinfo: -61
2022-12-01T11:13:32.6665044Z fi_getinfo: -61
2022-12-01T11:13:32.6665377Z fi_getinfo: -61
2022-12-01T11:13:32.6665631Z ok (7.187s)
2022-12-01T11:13:32.6665656Z 
2022-12-01T11:13:32.6666294Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6666567Z Ran 1 test in 7.188s
2022-12-01T11:13:32.6666613Z 
2022-12-01T11:13:32.6666825Z OK
2022-12-01T11:13:32.6666855Z 
2022-12-01T11:13:32.6667152Z Generating XML reports...
2022-12-01T11:13:32.6668610Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110409.xml
2022-12-01T11:13:32.6669589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6670014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6671007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6671493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6672130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6p79lnm9
2022-12-01T11:13:32.6672949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6p79lnm9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6672997Z 
2022-12-01T11:13:32.6673269Z Running tests...
2022-12-01T11:13:32.6674033Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6674982Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6675864Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6676398Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53089
2022-12-01T11:13:32.6676953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53090
2022-12-01T11:13:32.6677506Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53091
2022-12-01T11:13:32.6678067Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53092
2022-12-01T11:13:32.6679037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6679503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6680540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6681018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6681958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6682386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6683402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6683878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6684850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6685290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6686282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6686756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6687746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6688166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6689151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6689626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6690288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg186bmt4
2022-12-01T11:13:32.6690989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg186bmt4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6691661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmypi6h3
2022-12-01T11:13:32.6692365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmypi6h3/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6693030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyprot4zs
2022-12-01T11:13:32.6693705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyprot4zs/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6694327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpge6ls1_y
2022-12-01T11:13:32.6695560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpge6ls1_y/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6696154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6696888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6697492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6697988Z fi_getinfo: -61
2022-12-01T11:13:32.6698335Z fi_getinfo: -61
2022-12-01T11:13:32.6698885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6699230Z fi_getinfo: -61
2022-12-01T11:13:32.6699555Z fi_getinfo: -61
2022-12-01T11:13:32.6699782Z ok (9.785s)
2022-12-01T11:13:32.6699823Z 
2022-12-01T11:13:32.6700482Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6700754Z Ran 1 test in 9.785s
2022-12-01T11:13:32.6700786Z 
2022-12-01T11:13:32.6701002Z OK
2022-12-01T11:13:32.6701038Z 
2022-12-01T11:13:32.6701342Z Generating XML reports...
2022-12-01T11:13:32.6702812Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110418.xml
2022-12-01T11:13:32.6703804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6704246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6705390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6706145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6706938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu00qnj39
2022-12-01T11:13:32.6707722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu00qnj39/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6707765Z 
2022-12-01T11:13:32.6708172Z Running tests...
2022-12-01T11:13:32.6708821Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6709860Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6710738Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6711383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53448
2022-12-01T11:13:32.6712033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53449
2022-12-01T11:13:32.6712749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53450
2022-12-01T11:13:32.6713389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53451
2022-12-01T11:13:32.6714465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6714988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6715981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6716551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6717616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6718129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6719191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6719834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6720915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6721444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6722379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6723091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6724113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6724679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6725693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6726240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6727020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbef0lyrf
2022-12-01T11:13:32.6727655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbef0lyrf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6728014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcr5bb93
2022-12-01T11:13:32.6728271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcr5bb93/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6728563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotytzi1v
2022-12-01T11:13:32.6728873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotytzi1v/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6729161Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp206a0q2c
2022-12-01T11:13:32.6729478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp206a0q2c/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6729745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6730063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6730370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6730643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6730782Z fi_getinfo: -61
2022-12-01T11:13:32.6730963Z fi_getinfo: -61
2022-12-01T11:13:32.6731135Z fi_getinfo: -61
2022-12-01T11:13:32.6731325Z fi_getinfo: -61
2022-12-01T11:13:32.6731458Z ok (7.391s)
2022-12-01T11:13:32.6731483Z 
2022-12-01T11:13:32.6731785Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6731975Z Ran 1 test in 7.392s
2022-12-01T11:13:32.6731991Z 
2022-12-01T11:13:32.6732066Z OK
2022-12-01T11:13:32.6732083Z 
2022-12-01T11:13:32.6732246Z Generating XML reports...
2022-12-01T11:13:32.6732838Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110430.xml
2022-12-01T11:13:32.6733255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6733480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6733901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6734125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6734613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyektj3_4
2022-12-01T11:13:32.6734880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyektj3_4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6735014Z 
2022-12-01T11:13:32.6735114Z Running tests...
2022-12-01T11:13:32.6735457Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6735866Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6736234Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6736582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53799
2022-12-01T11:13:32.6736846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53800
2022-12-01T11:13:32.6737098Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53801
2022-12-01T11:13:32.6737415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53802
2022-12-01T11:13:32.6737778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6738041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6738473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6738706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6739114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6739328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6739745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6739976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6740327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6740574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6741030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6741261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6741672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6741885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6742297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6742530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6742821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3wo_72z
2022-12-01T11:13:32.6743077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3wo_72z/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6743386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzwhtx3qh
2022-12-01T11:13:32.6743706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8cqoqnb
2022-12-01T11:13:32.6744022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzwhtx3qh/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6744323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8cqoqnb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6744621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8gn_3hc
2022-12-01T11:13:32.6744920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8gn_3hc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6745191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6745469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6745684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6745983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6746198Z fi_getinfo: -61
2022-12-01T11:13:32.6746380Z fi_getinfo: -61
2022-12-01T11:13:32.6746550Z fi_getinfo: -61
2022-12-01T11:13:32.6746723Z fi_getinfo: -61
2022-12-01T11:13:32.6746863Z ok (7.249s)
2022-12-01T11:13:32.6746879Z 
2022-12-01T11:13:32.6747195Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6747366Z Ran 1 test in 7.249s
2022-12-01T11:13:32.6747389Z 
2022-12-01T11:13:32.6747513Z OK
2022-12-01T11:13:32.6747596Z 
2022-12-01T11:13:32.6747762Z Generating XML reports...
2022-12-01T11:13:32.6748400Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110440.xml
2022-12-01T11:13:32.6748824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6749034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6749464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6749646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6749937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29gscmz6
2022-12-01T11:13:32.6750269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29gscmz6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6750294Z 
2022-12-01T11:13:32.6750447Z Running tests...
2022-12-01T11:13:32.6750754Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6751188Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6751522Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6751798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54150
2022-12-01T11:13:32.6752055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54151
2022-12-01T11:13:32.6752255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54152
2022-12-01T11:13:32.6752507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54153
2022-12-01T11:13:32.6752929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6753149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6753569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6753830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6754247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6754463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6754821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6755049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6755456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6755697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6756112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6756337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6756794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6757000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6757413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6757580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6757932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdsq4x3b
2022-12-01T11:13:32.6758250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdsq4x3b/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6758600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0eii6wrs
2022-12-01T11:13:32.6758916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0eii6wrs/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6759207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeod6htyz
2022-12-01T11:13:32.6759556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeod6htyz/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6759852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuae8ejhg
2022-12-01T11:13:32.6760101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuae8ejhg/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6760373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6760671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6760943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6761217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6761407Z fi_getinfo: -61
2022-12-01T11:13:32.6761580Z fi_getinfo: -61
2022-12-01T11:13:32.6761789Z fi_getinfo: -61
2022-12-01T11:13:32.6761913Z fi_getinfo: -61
2022-12-01T11:13:32.6762054Z ok (9.706s)
2022-12-01T11:13:32.6762078Z 
2022-12-01T11:13:32.6762381Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6762524Z Ran 1 test in 9.706s
2022-12-01T11:13:32.6762549Z 
2022-12-01T11:13:32.6762691Z OK
2022-12-01T11:13:32.6762708Z 
2022-12-01T11:13:32.6762875Z Generating XML reports...
2022-12-01T11:13:32.6763468Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110449.xml
2022-12-01T11:13:32.6763883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6777456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6777972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6778182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6778439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6rd06m7
2022-12-01T11:13:32.6778719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6rd06m7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6778744Z 
2022-12-01T11:13:32.6778854Z Running tests...
2022-12-01T11:13:32.6779137Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6779533Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6779846Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6780084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54509
2022-12-01T11:13:32.6780315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54510
2022-12-01T11:13:32.6780529Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54511
2022-12-01T11:13:32.6780749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54512
2022-12-01T11:13:32.6781149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6781336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6781861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6782075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6782554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6782735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6783133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6783313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6783707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6783888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6784293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6784491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6784887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6785069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6785473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6785654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6785925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdehn9u91
2022-12-01T11:13:32.6786206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdehn9u91/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6786472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbcyd_ns
2022-12-01T11:13:32.6786750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbcyd_ns/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6787011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi44b_rrb
2022-12-01T11:13:32.6787294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi44b_rrb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6787565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppgjgiitw
2022-12-01T11:13:32.6787843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppgjgiitw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6788066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6788303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6788545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6788785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6788941Z fi_getinfo: -61
2022-12-01T11:13:32.6789088Z fi_getinfo: -61
2022-12-01T11:13:32.6789228Z fi_getinfo: -61
2022-12-01T11:13:32.6789368Z fi_getinfo: -61
2022-12-01T11:13:32.6789467Z ok (9.720s)
2022-12-01T11:13:32.6789484Z 
2022-12-01T11:13:32.6789763Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6789880Z Ran 1 test in 9.721s
2022-12-01T11:13:32.6789904Z 
2022-12-01T11:13:32.6789986Z OK
2022-12-01T11:13:32.6790010Z 
2022-12-01T11:13:32.6790133Z Generating XML reports...
2022-12-01T11:13:32.6790715Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110501.xml
2022-12-01T11:13:32.6791108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6791290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6791732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6791946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6792266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpo2gz_e
2022-12-01T11:13:32.6792549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpo2gz_e/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6792565Z 
2022-12-01T11:13:32.6792672Z Running tests...
2022-12-01T11:13:32.6792959Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6793341Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6793681Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6793895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54868
2022-12-01T11:13:32.6794127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54869
2022-12-01T11:13:32.6794362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54870
2022-12-01T11:13:32.6794592Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54871
2022-12-01T11:13:32.6794984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6795165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6795573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6795771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6796158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6796333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6796732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6796931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6797322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6797508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6797902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6798101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6798490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6798662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6799067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6799274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6799537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxk95g9od
2022-12-01T11:13:32.6799860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxk95g9od/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6800081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37idrqms
2022-12-01T11:13:32.6800360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37idrqms/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6800616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjanqix4m
2022-12-01T11:13:32.6800956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjanqix4m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6801211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsj2haz6b
2022-12-01T11:13:32.6801487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsj2haz6b/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6801790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6802030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6802268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6802514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6802661Z fi_getinfo: -61
2022-12-01T11:13:32.6802808Z fi_getinfo: -61
2022-12-01T11:13:32.6802940Z fi_getinfo: -61
2022-12-01T11:13:32.6803080Z fi_getinfo: -61
2022-12-01T11:13:32.6803178Z ok (4.086s)
2022-12-01T11:13:32.6803202Z 
2022-12-01T11:13:32.6803480Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6803604Z Ran 1 test in 4.087s
2022-12-01T11:13:32.6803621Z 
2022-12-01T11:13:32.6803712Z OK
2022-12-01T11:13:32.6803737Z 
2022-12-01T11:13:32.6803861Z Generating XML reports...
2022-12-01T11:13:32.6804423Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110513.xml
2022-12-01T11:13:32.6804818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6804998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6805399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6805603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6805869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1lxp2rx
2022-12-01T11:13:32.6806156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1lxp2rx/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6806174Z 
2022-12-01T11:13:32.6806289Z Running tests...
2022-12-01T11:13:32.6806627Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6806989Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6807335Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6807558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55055
2022-12-01T11:13:32.6807788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55056
2022-12-01T11:13:32.6808016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55057
2022-12-01T11:13:32.6808246Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55058
2022-12-01T11:13:32.6808645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6808833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6809236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6809419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6809813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6809982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6810386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6810726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6811038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6811291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6811681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6811861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6812251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6812431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6812830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6813026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6813296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqosdd308
2022-12-01T11:13:32.6813575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqosdd308/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6813848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga9v4870
2022-12-01T11:13:32.6814124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga9v4870/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6814587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp802sf7zf
2022-12-01T11:13:32.6814873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp802sf7zf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6815153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5trjicx
2022-12-01T11:13:32.6815410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5trjicx/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6815635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6815875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6816114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6816349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6816493Z fi_getinfo: -61
2022-12-01T11:13:32.6816635Z fi_getinfo: -61
2022-12-01T11:13:32.6816779Z fi_getinfo: -61
2022-12-01T11:13:32.6816924Z fi_getinfo: -61
2022-12-01T11:13:32.6817026Z ok (4.008s)
2022-12-01T11:13:32.6817047Z 
2022-12-01T11:13:32.6817330Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6817442Z Ran 1 test in 4.008s
2022-12-01T11:13:32.6817463Z 
2022-12-01T11:13:32.6817537Z OK
2022-12-01T11:13:32.6817572Z 
2022-12-01T11:13:32.6817682Z Generating XML reports...
2022-12-01T11:13:32.6818262Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110519.xml
2022-12-01T11:13:32.6818658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6818845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6819251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6819448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6819716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinyiivil
2022-12-01T11:13:32.6819997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinyiivil/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6820017Z 
2022-12-01T11:13:32.6820110Z Running tests...
2022-12-01T11:13:32.6820474Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6820862Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6821255Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6821486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55242
2022-12-01T11:13:32.6821715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55243
2022-12-01T11:13:32.6821943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55244
2022-12-01T11:13:32.6822172Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55245
2022-12-01T11:13:32.6822572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6822736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6823137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6823332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6823728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6823911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6824315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6824508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6824899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6825065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6825467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6825661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6826050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6826231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6826630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6826825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6827093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4hwp_uqg
2022-12-01T11:13:32.6827372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4hwp_uqg/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6827625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvqc1ete
2022-12-01T11:13:32.6827904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvqc1ete/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6828167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjie19bmc
2022-12-01T11:13:32.6828447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjie19bmc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6828708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk_bjuog
2022-12-01T11:13:32.6828981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk_bjuog/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6829222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6829461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6829682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6829886Z fi_getinfo: -61
2022-12-01T11:13:32.6830132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6830283Z fi_getinfo: -61
2022-12-01T11:13:32.6830484Z fi_getinfo: -61
2022-12-01T11:13:32.6830629Z fi_getinfo: -61
2022-12-01T11:13:32.6830733Z ok (3.989s)
2022-12-01T11:13:32.6830753Z 
2022-12-01T11:13:32.6831015Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6831131Z Ran 1 test in 3.989s
2022-12-01T11:13:32.6831151Z 
2022-12-01T11:13:32.6831351Z OK
2022-12-01T11:13:32.6831351Z 
2022-12-01T11:13:32.6831465Z Generating XML reports...
2022-12-01T11:13:32.6831961Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110526.xml
2022-12-01T11:13:32.6832357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6832547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6832953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6833156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6833406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamux0f0n
2022-12-01T11:13:32.6833687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamux0f0n/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6833707Z 
2022-12-01T11:13:32.6833815Z Running tests...
2022-12-01T11:13:32.6834093Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6834470Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6834786Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6835015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55417
2022-12-01T11:13:32.6835248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55418
2022-12-01T11:13:32.6835481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55419
2022-12-01T11:13:32.6835692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55420
2022-12-01T11:13:32.6836085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6836263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6836669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6836867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6837262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6837445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6837849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6838034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6838424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6838606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6839008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6839208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6839601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6839839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6840252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6840500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6840751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc9n6xh_
2022-12-01T11:13:32.6841032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc9n6xh_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6841296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ku1ul01
2022-12-01T11:13:32.6841573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ku1ul01/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6841834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppepdz0bu
2022-12-01T11:13:32.6842221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppepdz0bu/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6842480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0f7jyyt
2022-12-01T11:13:32.6842674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0f7jyyt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6842888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6843131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6843368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6843608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6843764Z fi_getinfo: -61
2022-12-01T11:13:32.6843910Z fi_getinfo: -61
2022-12-01T11:13:32.6844055Z fi_getinfo: -61
2022-12-01T11:13:32.6844198Z fi_getinfo: -61
2022-12-01T11:13:32.6844282Z ok (3.975s)
2022-12-01T11:13:32.6844305Z 
2022-12-01T11:13:32.6844588Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6844700Z Ran 1 test in 3.975s
2022-12-01T11:13:32.6844724Z 
2022-12-01T11:13:32.6844812Z OK
2022-12-01T11:13:32.6844831Z 
2022-12-01T11:13:32.6844957Z Generating XML reports...
2022-12-01T11:13:32.6845533Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110532.xml
2022-12-01T11:13:32.6845927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6846112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6846503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6846703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6846969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8reg_me
2022-12-01T11:13:32.6847246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8reg_me/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6847269Z 
2022-12-01T11:13:32.6847378Z Running tests...
2022-12-01T11:13:32.6847656Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6848034Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6848353Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6848584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55604
2022-12-01T11:13:32.6848798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55605
2022-12-01T11:13:32.6849026Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55606
2022-12-01T11:13:32.6849303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55607
2022-12-01T11:13:32.6849712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6849947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6850356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6850556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6850948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6851113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6851514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6851718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6852108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6852292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6852698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6852896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6853284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6853466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6853853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6854051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6854609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfa89nbf
2022-12-01T11:13:32.6854913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfa89nbf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6855172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqja989gt
2022-12-01T11:13:32.6855426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqja989gt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6855695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabochqsx
2022-12-01T11:13:32.6855978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabochqsx/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6856222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf5a2je3z
2022-12-01T11:13:32.6856499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf5a2je3z/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6856744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6856989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6857232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6857392Z fi_getinfo: -61
2022-12-01T11:13:32.6857541Z fi_getinfo: -61
2022-12-01T11:13:32.6857777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6857906Z fi_getinfo: -61
2022-12-01T11:13:32.6858050Z fi_getinfo: -61
2022-12-01T11:13:32.6858149Z ok (6.093s)
2022-12-01T11:13:32.6858168Z 
2022-12-01T11:13:32.6858448Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6858561Z Ran 1 test in 6.094s
2022-12-01T11:13:32.6858581Z 
2022-12-01T11:13:32.6858673Z OK
2022-12-01T11:13:32.6858692Z 
2022-12-01T11:13:32.6858820Z Generating XML reports...
2022-12-01T11:13:32.6859477Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110538.xml
2022-12-01T11:13:32.6859927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6860112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6860518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6860719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6860990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpha05yyjv
2022-12-01T11:13:32.6861266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpha05yyjv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6861287Z 
2022-12-01T11:13:32.6861396Z Running tests...
2022-12-01T11:13:32.6861675Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6862041Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6862375Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6862604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55951
2022-12-01T11:13:32.6862829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55952
2022-12-01T11:13:32.6863056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55953
2022-12-01T11:13:32.6863282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55954
2022-12-01T11:13:32.6863679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6863861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6864266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6864454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6864848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6865028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6865429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6865630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6866023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6866205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6866608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6866804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6867185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6867367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6867768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6867964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6868229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe5m3m_sa
2022-12-01T11:13:32.6868505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe5m3m_sa/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6868820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbombk8ze
2022-12-01T11:13:32.6869115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbombk8ze/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6869410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vyi3p6k
2022-12-01T11:13:32.6869691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vyi3p6k/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6869957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45gwsa96
2022-12-01T11:13:32.6870232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45gwsa96/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6870474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6870714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6870950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6871188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6871326Z fi_getinfo: -61
2022-12-01T11:13:32.6871477Z fi_getinfo: -61
2022-12-01T11:13:32.6871619Z fi_getinfo: -61
2022-12-01T11:13:32.6871760Z fi_getinfo: -61
2022-12-01T11:13:32.6871859Z ok (6.301s)
2022-12-01T11:13:32.6871879Z 
2022-12-01T11:13:32.6872158Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6872272Z Ran 1 test in 6.301s
2022-12-01T11:13:32.6872292Z 
2022-12-01T11:13:32.6872383Z OK
2022-12-01T11:13:32.6872402Z 
2022-12-01T11:13:32.6872512Z Generating XML reports...
2022-12-01T11:13:32.6873086Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110546.xml
2022-12-01T11:13:32.6873480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6873663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6874069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6874269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6874536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqar0mj2m
2022-12-01T11:13:32.6874813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqar0mj2m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6874833Z 
2022-12-01T11:13:32.6874939Z Running tests...
2022-12-01T11:13:32.6875203Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6875586Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6875927Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6876156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56298
2022-12-01T11:13:32.6876390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56299
2022-12-01T11:13:32.6876620Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56300
2022-12-01T11:13:32.6876846Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56301
2022-12-01T11:13:32.6877236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6877403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6877806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6878001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6878437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6878625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6879097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6879292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6879744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6879926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6880311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6880508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6880902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6881080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6881479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6881681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6881947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2ey81my
2022-12-01T11:13:32.6882223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2ey81my/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6882473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzuzptl16
2022-12-01T11:13:32.6882754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzuzptl16/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6883014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ch9a70s
2022-12-01T11:13:32.6883289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ch9a70s/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6883553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnaqmce7x
2022-12-01T11:13:32.6883832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnaqmce7x/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6884071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6884310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6884549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6884772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6884924Z fi_getinfo: -61
2022-12-01T11:13:32.6885069Z fi_getinfo: -61
2022-12-01T11:13:32.6885215Z fi_getinfo: -61
2022-12-01T11:13:32.6885359Z fi_getinfo: -61
2022-12-01T11:13:32.6885464Z ok (6.109s)
2022-12-01T11:13:32.6885485Z 
2022-12-01T11:13:32.6885764Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6885866Z Ran 1 test in 6.109s
2022-12-01T11:13:32.6885886Z 
2022-12-01T11:13:32.6885974Z OK
2022-12-01T11:13:32.6885994Z 
2022-12-01T11:13:32.6886116Z Generating XML reports...
2022-12-01T11:13:32.6886689Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110555.xml
2022-12-01T11:13:32.6887083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6887263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6887664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6887862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6888164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsw_4edj
2022-12-01T11:13:32.6888453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsw_4edj/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6888516Z 
2022-12-01T11:13:32.6888631Z Running tests...
2022-12-01T11:13:32.6888946Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6889317Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6889641Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6889861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56645
2022-12-01T11:13:32.6890080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56646
2022-12-01T11:13:32.6890302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56647
2022-12-01T11:13:32.6890497Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56648
2022-12-01T11:13:32.6890876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6891055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6891436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6891628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6891994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6892167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6892537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6892725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6893068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6893244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6893612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6893797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6894159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6894515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6894906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6895116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6895320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpno0ysym1
2022-12-01T11:13:32.6895596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpno0ysym1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6895849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeopqp3dl
2022-12-01T11:13:32.6896118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeopqp3dl/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6896372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf9_e4jv
2022-12-01T11:13:32.6896637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf9_e4jv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6896885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcb6wsp2
2022-12-01T11:13:32.6897219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcb6wsp2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6897461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6897731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6897960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6898187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6898331Z fi_getinfo: -61
2022-12-01T11:13:32.6898471Z fi_getinfo: -61
2022-12-01T11:13:32.6898603Z fi_getinfo: -61
2022-12-01T11:13:32.6898735Z fi_getinfo: -61
2022-12-01T11:13:32.6898817Z ok (6.090s)
2022-12-01T11:13:32.6898837Z 
2022-12-01T11:13:32.6899104Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6899215Z Ran 1 test in 6.090s
2022-12-01T11:13:32.6899235Z 
2022-12-01T11:13:32.6899325Z OK
2022-12-01T11:13:32.6899344Z 
2022-12-01T11:13:32.6899472Z Generating XML reports...
2022-12-01T11:13:32.6900025Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110603.xml
2022-12-01T11:13:32.6900402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6900579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6900946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6901139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6901395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpniu4g52p
2022-12-01T11:13:32.6901664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpniu4g52p/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6901684Z 
2022-12-01T11:13:32.6901789Z Running tests...
2022-12-01T11:13:32.6902056Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6902413Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6902751Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6902969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56992
2022-12-01T11:13:32.6903170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56993
2022-12-01T11:13:32.6903379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56994
2022-12-01T11:13:32.6903588Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56995
2022-12-01T11:13:32.6903962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6904139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6904520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6904712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6905075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6905247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6905604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6905792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6906147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6906373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6906802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6907144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6907413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6907584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6907938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6908124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6908379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ct6whxy
2022-12-01T11:13:32.6908648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ct6whxy/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6908907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4r3jhudm
2022-12-01T11:13:32.6909171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4r3jhudm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6909429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0pi0avbp
2022-12-01T11:13:32.6909697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0pi0avbp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6909947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppapgf6nf
2022-12-01T11:13:32.6910199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppapgf6nf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6910424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6910651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6910878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6911103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6911254Z fi_getinfo: -61
2022-12-01T11:13:32.6911388Z fi_getinfo: -61
2022-12-01T11:13:32.6911505Z fi_getinfo: -61
2022-12-01T11:13:32.6911639Z fi_getinfo: -61
2022-12-01T11:13:32.6911736Z ok (6.118s)
2022-12-01T11:13:32.6911756Z 
2022-12-01T11:13:32.6912021Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6912132Z Ran 1 test in 6.118s
2022-12-01T11:13:32.6912152Z 
2022-12-01T11:13:32.6912240Z OK
2022-12-01T11:13:32.6912260Z 
2022-12-01T11:13:32.6912380Z Generating XML reports...
2022-12-01T11:13:32.6912928Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110612.xml
2022-12-01T11:13:32.6913289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6913465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6913845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6914039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6914292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6647hb6
2022-12-01T11:13:32.6914558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6647hb6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6914579Z 
2022-12-01T11:13:32.6914684Z Running tests...
2022-12-01T11:13:32.6914945Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6915300Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6915656Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6915884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57339
2022-12-01T11:13:32.6916149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57340
2022-12-01T11:13:32.6916366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57341
2022-12-01T11:13:32.6916577Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57342
2022-12-01T11:13:32.6916951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6917125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6917491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6917648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6918034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6918224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6918602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6918788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6919158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6919330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6919700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6919885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6920325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6920432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6920780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6920964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6921221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8dr4vob
2022-12-01T11:13:32.6921489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8dr4vob/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6921739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhiolday
2022-12-01T11:13:32.6922004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhiolday/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6922260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhcygzqh
2022-12-01T11:13:32.6922512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhcygzqh/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6922763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4surwjic
2022-12-01T11:13:32.6923029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4surwjic/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6923255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6923481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6923801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6923943Z fi_getinfo: -61
2022-12-01T11:13:32.6924082Z fi_getinfo: -61
2022-12-01T11:13:32.6924282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6924412Z fi_getinfo: -61
2022-12-01T11:13:32.6924513Z fi_getinfo: -61
2022-12-01T11:13:32.6924620Z ok (6.084s)
2022-12-01T11:13:32.6924640Z 
2022-12-01T11:13:32.6924907Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6925067Z Ran 1 test in 6.085s
2022-12-01T11:13:32.6925087Z 
2022-12-01T11:13:32.6925172Z OK
2022-12-01T11:13:32.6925191Z 
2022-12-01T11:13:32.6925297Z Generating XML reports...
2022-12-01T11:13:32.6925850Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110620.xml
2022-12-01T11:13:32.6926223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6926394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6926768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6926962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6927214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1rboe6x
2022-12-01T11:13:32.6927486Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1rboe6x/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6927507Z 
2022-12-01T11:13:32.6927612Z Running tests...
2022-12-01T11:13:32.6927860Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6928222Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6928550Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6928769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57686
2022-12-01T11:13:32.6928991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57687
2022-12-01T11:13:32.6929210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57688
2022-12-01T11:13:32.6929423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57689
2022-12-01T11:13:32.6929799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6929973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6930337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6930526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6930891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6931063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6931436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6931621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6931985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6932153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6932516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6932701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6933056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6933227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6933647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6933844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6934100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3t3lopsw
2022-12-01T11:13:32.6934616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3t3lopsw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6934875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9yal1xb
2022-12-01T11:13:32.6935123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9yal1xb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6935341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfaj9gy1
2022-12-01T11:13:32.6935606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfaj9gy1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6935862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijw3knvr
2022-12-01T11:13:32.6936130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijw3knvr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6936361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6936592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6936812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6937021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6937171Z fi_getinfo: -61
2022-12-01T11:13:32.6937308Z fi_getinfo: -61
2022-12-01T11:13:32.6937442Z fi_getinfo: -61
2022-12-01T11:13:32.6937573Z fi_getinfo: -61
2022-12-01T11:13:32.6937675Z ok (6.198s)
2022-12-01T11:13:32.6937695Z 
2022-12-01T11:13:32.6937959Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6938052Z Ran 1 test in 6.199s
2022-12-01T11:13:32.6938088Z 
2022-12-01T11:13:32.6938166Z OK
2022-12-01T11:13:32.6938185Z 
2022-12-01T11:13:32.6938302Z Generating XML reports...
2022-12-01T11:13:32.6938852Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110628.xml
2022-12-01T11:13:32.6939231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6939408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6939789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6939976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6940227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3xav4n2g
2022-12-01T11:13:32.6940476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3xav4n2g/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6940512Z 
2022-12-01T11:13:32.6940604Z Running tests...
2022-12-01T11:13:32.6940868Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6941229Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6941526Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6941743Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58033
2022-12-01T11:13:32.6941959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58034
2022-12-01T11:13:32.6942169Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58035
2022-12-01T11:13:32.6942366Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58036
2022-12-01T11:13:32.6942813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6943002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6943385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6943655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6944028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6944200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6944568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6944756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6945099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6945279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6945647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6945834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6946193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6946366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6946741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6946924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6947166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo785enp
2022-12-01T11:13:32.6947440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo785enp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6947690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2o5tc82h
2022-12-01T11:13:32.6947958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2o5tc82h/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6948210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0d7sfb6
2022-12-01T11:13:32.6948476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0d7sfb6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6948723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fwbtypb
2022-12-01T11:13:32.6948984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fwbtypb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6949211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6949415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6949644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6949870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6950017Z fi_getinfo: -61
2022-12-01T11:13:32.6950156Z fi_getinfo: -61
2022-12-01T11:13:32.6950292Z fi_getinfo: -61
2022-12-01T11:13:32.6950425Z fi_getinfo: -61
2022-12-01T11:13:32.6950506Z ok (9.918s)
2022-12-01T11:13:32.6950541Z 
2022-12-01T11:13:32.6950790Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6950900Z Ran 1 test in 9.918s
2022-12-01T11:13:32.6950919Z 
2022-12-01T11:13:32.6951006Z OK
2022-12-01T11:13:32.6951025Z 
2022-12-01T11:13:32.6951145Z Generating XML reports...
2022-12-01T11:13:32.6951695Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110637.xml
2022-12-01T11:13:32.6952137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6952324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6952840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6953014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6953252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3vj9iri
2022-12-01T11:13:32.6953509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3vj9iri/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6953529Z 
2022-12-01T11:13:32.6953635Z Running tests...
2022-12-01T11:13:32.6953899Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6954419Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6954732Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6954953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58392
2022-12-01T11:13:32.6955177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58393
2022-12-01T11:13:32.6955374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58394
2022-12-01T11:13:32.6955593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58395
2022-12-01T11:13:32.6955965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6956137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6956517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6956709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6957076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6957254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6957607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6957794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6958152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6958321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6958693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6958874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6959232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6959400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6959780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6959948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6960204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0laoen2p
2022-12-01T11:13:32.6960476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0laoen2p/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6960728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cfkzr0b
2022-12-01T11:13:32.6960980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkltziql_
2022-12-01T11:13:32.6961298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cfkzr0b/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6961572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkltziql_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6961870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hxtijmq
2022-12-01T11:13:32.6962119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hxtijmq/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6962350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6962575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6962799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6963025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6963172Z fi_getinfo: -61
2022-12-01T11:13:32.6963308Z fi_getinfo: -61
2022-12-01T11:13:32.6963447Z fi_getinfo: -61
2022-12-01T11:13:32.6963566Z fi_getinfo: -61
2022-12-01T11:13:32.6963663Z ok (9.690s)
2022-12-01T11:13:32.6963683Z 
2022-12-01T11:13:32.6963954Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6964064Z Ran 1 test in 9.690s
2022-12-01T11:13:32.6964084Z 
2022-12-01T11:13:32.6964174Z OK
2022-12-01T11:13:32.6964193Z 
2022-12-01T11:13:32.6964316Z Generating XML reports...
2022-12-01T11:13:32.6964868Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110649.xml
2022-12-01T11:13:32.6965239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6965397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6965772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6965971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6966226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7q3gi0a
2022-12-01T11:13:32.6966495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7q3gi0a/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6966515Z 
2022-12-01T11:13:32.6966622Z Running tests...
2022-12-01T11:13:32.6966885Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6967243Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6967546Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6967748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58743
2022-12-01T11:13:32.6967963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58744
2022-12-01T11:13:32.6968178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58745
2022-12-01T11:13:32.6968389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58746
2022-12-01T11:13:32.6968765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6968939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6969315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6969502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6969849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6970019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6970437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6970628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6971037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6971210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6971579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6971766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6972132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6972287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6972664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6972847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6973099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6iaa8d3b
2022-12-01T11:13:32.6973373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6iaa8d3b/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6973626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv0691lu
2022-12-01T11:13:32.6973890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv0691lu/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6974140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbto5q01j
2022-12-01T11:13:32.6974670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbto5q01j/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6974950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisliyvbn
2022-12-01T11:13:32.6975216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisliyvbn/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6975418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6975642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6975868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6976093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6976246Z fi_getinfo: -61
2022-12-01T11:13:32.6976330Z ok (3.910s)
2022-12-01T11:13:32.6976365Z 
2022-12-01T11:13:32.6976614Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6976726Z Ran 1 test in 3.910s
2022-12-01T11:13:32.6976746Z 
2022-12-01T11:13:32.6976833Z OK
2022-12-01T11:13:32.6976853Z 
2022-12-01T11:13:32.6976972Z Generating XML reports...
2022-12-01T11:13:32.6977521Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110701.xml
2022-12-01T11:13:32.6977900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6978078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6978454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6978628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6978878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v44iv80
2022-12-01T11:13:32.6979144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v44iv80/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6979163Z 
2022-12-01T11:13:32.6979269Z Running tests...
2022-12-01T11:13:32.6979636Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6980012Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6980377Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6980599Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58915
2022-12-01T11:13:32.6980798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58916
2022-12-01T11:13:32.6981016Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58917
2022-12-01T11:13:32.6981230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58918
2022-12-01T11:13:32.6981604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6981780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6982165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6982359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6982735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6982910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6983267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6983456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6983820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6983995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6984375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6984565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6984931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6985104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6985460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6985651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6985904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpba3mbm5b
2022-12-01T11:13:32.6986175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpba3mbm5b/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6986427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp152io7b7
2022-12-01T11:13:32.6986691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp152io7b7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6986946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnilhe6d0
2022-12-01T11:13:32.6987210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnilhe6d0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6987453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwlpovu3
2022-12-01T11:13:32.6987703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwlpovu3/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6987928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.6988147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.6988369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.6988643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.6988801Z fi_getinfo: -61
2022-12-01T11:13:32.6988989Z fi_getinfo: -61
2022-12-01T11:13:32.6989125Z fi_getinfo: -61
2022-12-01T11:13:32.6989243Z fi_getinfo: -61
2022-12-01T11:13:32.6989342Z ok (9.708s)
2022-12-01T11:13:32.6989361Z 
2022-12-01T11:13:32.6989625Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6989732Z Ran 1 test in 9.708s
2022-12-01T11:13:32.6989752Z 
2022-12-01T11:13:32.6989840Z OK
2022-12-01T11:13:32.6989859Z 
2022-12-01T11:13:32.6989978Z Generating XML reports...
2022-12-01T11:13:32.6990527Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110707.xml
2022-12-01T11:13:32.6990899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6991063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6991446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6991642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6991893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphld65fe5
2022-12-01T11:13:32.6992161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphld65fe5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6992181Z 
2022-12-01T11:13:32.6992289Z Running tests...
2022-12-01T11:13:32.6992553Z ----------------------------------------------------------------------
2022-12-01T11:13:32.6992914Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.6993206Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.6993426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59274
2022-12-01T11:13:32.6993646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59275
2022-12-01T11:13:32.6993863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59276
2022-12-01T11:13:32.6994073Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59277
2022-12-01T11:13:32.6994445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6994617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6994998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6995189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6995546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6995719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6996088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6996261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6996635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6996824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6997199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6997387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6997736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.6997963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.6998355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.6998592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.6998849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fc6du55
2022-12-01T11:13:32.6999119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fc6du55/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6999374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqirz_01
2022-12-01T11:13:32.6999639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqirz_01/_remote_module_non_scriptable.py
2022-12-01T11:13:32.6999893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphlsy99z
2022-12-01T11:13:32.7000148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphlsy99z/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7000399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl47jg8i9
2022-12-01T11:13:32.7000668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl47jg8i9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7000895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7001118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7001345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7001565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7001710Z skip: Need at least 4 CUDA devices (3.759s)
2022-12-01T11:13:32.7001730Z 
2022-12-01T11:13:32.7001998Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7002097Z Ran 1 test in 3.759s
2022-12-01T11:13:32.7002117Z 
2022-12-01T11:13:32.7002225Z OK (skipped=1)
2022-12-01T11:13:32.7002244Z 
2022-12-01T11:13:32.7002366Z Generating XML reports...
2022-12-01T11:13:32.7002920Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110719.xml
2022-12-01T11:13:32.7003291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7003468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7003845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7004036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7004271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6i16div8
2022-12-01T11:13:32.7004540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6i16div8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7004561Z 
2022-12-01T11:13:32.7004668Z Running tests...
2022-12-01T11:13:32.7004938Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7005294Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7005608Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7005826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59445
2022-12-01T11:13:32.7006045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59446
2022-12-01T11:13:32.7006260Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59447
2022-12-01T11:13:32.7006455Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59448
2022-12-01T11:13:32.7006920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7007104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7007553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7007747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7008118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7008291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7008663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7008834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7009202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7009375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7009747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7009934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7010295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7010467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7010842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7011027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7011266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcw5czfoe
2022-12-01T11:13:32.7011538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcw5czfoe/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7011791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxqus08d
2022-12-01T11:13:32.7012062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxqus08d/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7012314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohjmwxvm
2022-12-01T11:13:32.7012585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohjmwxvm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7012836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpur3umgr6
2022-12-01T11:13:32.7013100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpur3umgr6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7013310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7013533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7013759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7013987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7014135Z skip: Need at least 4 CUDA devices (3.888s)
2022-12-01T11:13:32.7014156Z 
2022-12-01T11:13:32.7014659Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7014778Z Ran 1 test in 3.888s
2022-12-01T11:13:32.7014799Z 
2022-12-01T11:13:32.7014887Z OK (skipped=1)
2022-12-01T11:13:32.7014913Z 
2022-12-01T11:13:32.7015035Z Generating XML reports...
2022-12-01T11:13:32.7015550Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110725.xml
2022-12-01T11:13:32.7015997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7016185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7016631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7016824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7017080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxfif5ds
2022-12-01T11:13:32.7017352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxfif5ds/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7017372Z 
2022-12-01T11:13:32.7017480Z Running tests...
2022-12-01T11:13:32.7017746Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7018088Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7018403Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7018621Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59616
2022-12-01T11:13:32.7018843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59617
2022-12-01T11:13:32.7019056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59618
2022-12-01T11:13:32.7019271Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59619
2022-12-01T11:13:32.7019645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7019821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7020186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7020379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7020752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7021032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7021322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7021498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7021876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7022066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7022443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7022615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7022983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7023155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7023537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7023725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7023983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s0w16jv
2022-12-01T11:13:32.7024253Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s0w16jv/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7024506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptuaai5v
2022-12-01T11:13:32.7024757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptuaai5v/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7025060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc6zjt37
2022-12-01T11:13:32.7025331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc6zjt37/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7025629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpump_hp4_
2022-12-01T11:13:32.7025889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpump_hp4_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7026118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7026342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7026569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7026793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7026925Z fi_getinfo: -61
2022-12-01T11:13:32.7027063Z fi_getinfo: -61
2022-12-01T11:13:32.7027202Z fi_getinfo: -61
2022-12-01T11:13:32.7027339Z fi_getinfo: -61
2022-12-01T11:13:32.7027438Z ok (4.100s)
2022-12-01T11:13:32.7027457Z 
2022-12-01T11:13:32.7027721Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7027835Z Ran 1 test in 4.100s
2022-12-01T11:13:32.7027854Z 
2022-12-01T11:13:32.7027927Z OK
2022-12-01T11:13:32.7027946Z 
2022-12-01T11:13:32.7028069Z Generating XML reports...
2022-12-01T11:13:32.7028622Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110731.xml
2022-12-01T11:13:32.7028996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7029171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7029549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7029743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7029999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpug3vootw
2022-12-01T11:13:32.7030270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpug3vootw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7030289Z 
2022-12-01T11:13:32.7030380Z Running tests...
2022-12-01T11:13:32.7030645Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7031004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7031298Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7031516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59803
2022-12-01T11:13:32.7031728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59804
2022-12-01T11:13:32.7031943Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59805
2022-12-01T11:13:32.7032155Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59806
2022-12-01T11:13:32.7032518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7032693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7033069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7033261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7033631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7033803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7034228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7034423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7034840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7034997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7035370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7035555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7035917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7036090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7036468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7036660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7036917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy7awjco
2022-12-01T11:13:32.7037177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy7awjco/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7037431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlrngzs6
2022-12-01T11:13:32.7037700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlrngzs6/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7037953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp_89qz_
2022-12-01T11:13:32.7038216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp_89qz_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7038466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplcdjbabc
2022-12-01T11:13:32.7038736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplcdjbabc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7038961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7039184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7039394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7039618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7039764Z fi_getinfo: -61
2022-12-01T11:13:32.7039901Z fi_getinfo: -61
2022-12-01T11:13:32.7040032Z fi_getinfo: -61
2022-12-01T11:13:32.7040165Z fi_getinfo: -61
2022-12-01T11:13:32.7040298Z On WorkerInfo(id=1, name=worker1):
2022-12-01T11:13:32.7052496Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7faa73db483b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7faa73db020e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7faa7e91d0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7faa7e91e5af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7faa7e91fdb2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7faa7ec0953e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x297c70e (0x7faa7697370e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: <unknown function> + 0x297c816 (0x7faa76973816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7faa7f6afde8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x34c2400 (0x7faa80eef400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x34c2b79 (0x7faa80eefb79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7faa7f6e80e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2f75a4 (0x7faa8bc065a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2f7996 (0x7faa8bc06996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55dc3d235c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55dc3d1f1499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55dc3d1f15fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55dc3d19d4b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55dc3d23a098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55dc3d1e7742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55dc3d19ffaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55dc3d23b774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55dc3d1e7742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55dc3d19ffaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xa8449f (0x7faa8c39349f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7faa8c3916cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7faa8c3949c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7faa8c396206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7faa8233cb8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7faa8c3947b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4908e23 (0x7faa82335e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7faa823369b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7faa82330af7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x49387d2 (0x7faa823657d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7faa73da2c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7faaa3a7dbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7faac40d16db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7faac3dfa61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-12-01T11:13:32.7052773Z Traceback (most recent call last):
2022-12-01T11:13:32.7053149Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-12-01T11:13:32.7053342Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-12-01T11:13:32.7053752Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5911, in _gpu_add_wrong_gpus
2022-12-01T11:13:32.7053872Z     return x.cpu() + y.cuda()
2022-12-01T11:13:32.7054120Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-12-01T11:13:32.7054607Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-12-01T11:13:32.7055194Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7faa73db483b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7055782Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7faa73db020e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7056293Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7faa7e91d0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7056762Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7faa7e91e5af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7057343Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7faa7e91fdb2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7057869Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7faa7ec0953e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7058258Z frame #6: <unknown function> + 0x297c70e (0x7faa7697370e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7058639Z frame #7: <unknown function> + 0x297c816 (0x7faa76973816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7059194Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7faa7f6afde8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7059572Z frame #9: <unknown function> + 0x34c2400 (0x7faa80eef400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7059954Z frame #10: <unknown function> + 0x34c2b79 (0x7faa80eefb79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7060528Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7faa7f6e80e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7060937Z frame #12: <unknown function> + 0x2f75a4 (0x7faa8bc065a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7061376Z frame #13: <unknown function> + 0x2f7996 (0x7faa8bc06996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7061578Z frame #14: <unknown function> + 0x1ddc68 (0x55dc3d235c68 in /opt/conda/bin/python)
2022-12-01T11:13:32.7061765Z frame #15: <unknown function> + 0x199499 (0x55dc3d1f1499 in /opt/conda/bin/python)
2022-12-01T11:13:32.7061950Z frame #16: <unknown function> + 0x1995fa (0x55dc3d1f15fa in /opt/conda/bin/python)
2022-12-01T11:13:32.7062136Z frame #17: PyNumber_Add + 0x41 (0x55dc3d19d4b1 in /opt/conda/bin/python)
2022-12-01T11:13:32.7062345Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55dc3d23a098 in /opt/conda/bin/python)
2022-12-01T11:13:32.7062542Z frame #19: <unknown function> + 0x18f742 (0x55dc3d1e7742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7062732Z frame #20: _PyObject_Call + 0x20a (0x55dc3d19ffaa in /opt/conda/bin/python)
2022-12-01T11:13:32.7062929Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55dc3d23b774 in /opt/conda/bin/python)
2022-12-01T11:13:32.7063117Z frame #22: <unknown function> + 0x18f742 (0x55dc3d1e7742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7063302Z frame #23: _PyObject_Call + 0x20a (0x55dc3d19ffaa in /opt/conda/bin/python)
2022-12-01T11:13:32.7063693Z frame #24: <unknown function> + 0xa8449f (0x7faa8c39349f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7064219Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7faa8c3916cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7064883Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7faa8c3949c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7065570Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7faa8c396206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7066336Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7faa8233cb8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7067122Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7faa8c3947b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7067504Z frame #30: <unknown function> + 0x4908e23 (0x7faa82335e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7068175Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7faa823369b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7068800Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7faa82330af7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7069178Z frame #33: <unknown function> + 0x49387d2 (0x7faa823657d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7069648Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7faa73da2c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7069918Z frame #35: <unknown function> + 0xdbbf4 (0x7faaa3a7dbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-12-01T11:13:32.7070248Z frame #36: <unknown function> + 0x76db (0x7faac40d16db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-12-01T11:13:32.7070519Z frame #37: clone + 0x3f (0x7faac3dfa61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-12-01T11:13:32.7070540Z 
2022-12-01T11:13:32.7070559Z 
2022-12-01T11:13:32.7070696Z On WorkerInfo(id=0, name=worker0):
2022-12-01T11:13:32.7082783Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fb84569183b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fb84568d20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7fb8501fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fb8501fb5af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fb8501fcdb2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fb8504e653e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x297c70e (0x7fb84825070e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: <unknown function> + 0x297c816 (0x7fb848250816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fb850f8cde8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x34c2400 (0x7fb8527cc400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x34c2b79 (0x7fb8527ccb79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fb850fc50e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2f75a4 (0x7fb85d4e35a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2f7996 (0x7fb85d4e3996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55fe6a7acc68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55fe6a768499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55fe6a7685fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55fe6a7144b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55fe6a7b1098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55fe6a75e742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55fe6a716faa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55fe6a7b2774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55fe6a75e742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55fe6a716faa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xa8449f (0x7fb85dc7049f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fb85dc6e6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fb85dc719c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fb85dc73206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fb853c19b8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fb85dc717b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4908e23 (0x7fb853c12e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fb853c139b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fb853c0daf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x49387d2 (0x7fb853c427d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fb84567fc6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7fb87535abf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7fb8959ae6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7fb8956d761f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-12-01T11:13:32.7083028Z Traceback (most recent call last):
2022-12-01T11:13:32.7083395Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-12-01T11:13:32.7083588Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-12-01T11:13:32.7083992Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5911, in _gpu_add_wrong_gpus
2022-12-01T11:13:32.7084103Z     return x.cpu() + y.cuda()
2022-12-01T11:13:32.7084349Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-12-01T11:13:32.7084638Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-12-01T11:13:32.7085197Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fb84569183b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7085818Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fb84568d20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7086363Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7fb8501fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7086880Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fb8501fb5af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7087458Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fb8501fcdb2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7087973Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fb8504e653e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7088360Z frame #6: <unknown function> + 0x297c70e (0x7fb84825070e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7088733Z frame #7: <unknown function> + 0x297c816 (0x7fb848250816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7089278Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fb850f8cde8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7089653Z frame #9: <unknown function> + 0x34c2400 (0x7fb8527cc400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7090030Z frame #10: <unknown function> + 0x34c2b79 (0x7fb8527ccb79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7090524Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fb850fc50e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7090915Z frame #12: <unknown function> + 0x2f75a4 (0x7fb85d4e35a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7091304Z frame #13: <unknown function> + 0x2f7996 (0x7fb85d4e3996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7091506Z frame #14: <unknown function> + 0x1ddc68 (0x55fe6a7acc68 in /opt/conda/bin/python)
2022-12-01T11:13:32.7091690Z frame #15: <unknown function> + 0x199499 (0x55fe6a768499 in /opt/conda/bin/python)
2022-12-01T11:13:32.7091874Z frame #16: <unknown function> + 0x1995fa (0x55fe6a7685fa in /opt/conda/bin/python)
2022-12-01T11:13:32.7092057Z frame #17: PyNumber_Add + 0x41 (0x55fe6a7144b1 in /opt/conda/bin/python)
2022-12-01T11:13:32.7092247Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55fe6a7b1098 in /opt/conda/bin/python)
2022-12-01T11:13:32.7092437Z frame #19: <unknown function> + 0x18f742 (0x55fe6a75e742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7092620Z frame #20: _PyObject_Call + 0x20a (0x55fe6a716faa in /opt/conda/bin/python)
2022-12-01T11:13:32.7092828Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55fe6a7b2774 in /opt/conda/bin/python)
2022-12-01T11:13:32.7093014Z frame #22: <unknown function> + 0x18f742 (0x55fe6a75e742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7093197Z frame #23: _PyObject_Call + 0x20a (0x55fe6a716faa in /opt/conda/bin/python)
2022-12-01T11:13:32.7093590Z frame #24: <unknown function> + 0xa8449f (0x7fb85dc7049f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7094117Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fb85dc6e6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7095029Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fb85dc719c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7095803Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fb85dc73206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7096639Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fb853c19b8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7097424Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fb85dc717b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7097793Z frame #30: <unknown function> + 0x4908e23 (0x7fb853c12e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7098459Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fb853c139b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7099094Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fb853c0daf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7099476Z frame #33: <unknown function> + 0x49387d2 (0x7fb853c427d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7099897Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fb84567fc6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7100120Z frame #35: <unknown function> + 0xdbbf4 (0x7fb87535abf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-12-01T11:13:32.7100444Z frame #36: <unknown function> + 0x76db (0x7fb8959ae6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-12-01T11:13:32.7100713Z frame #37: clone + 0x3f (0x7fb8956d761f in /lib/x86_64-linux-gnu/libc.so.6)
2022-12-01T11:13:32.7100734Z 
2022-12-01T11:13:32.7100753Z 
2022-12-01T11:13:32.7100886Z On WorkerInfo(id=3, name=worker3):
2022-12-01T11:13:32.7112973Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa87eeb683b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa87eeb220e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7fa889a1f0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fa889a205af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fa889a21db2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fa889d0b53e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x297c70e (0x7fa881a7570e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: <unknown function> + 0x297c816 (0x7fa881a75816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fa88a7b1de8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x34c2400 (0x7fa88bff1400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x34c2b79 (0x7fa88bff1b79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fa88a7ea0e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2f75a4 (0x7fa896d085a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2f7996 (0x7fa896d08996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55761522dc68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x5576151e9499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x5576151e95fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x5576151954b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x557615232098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x5576151df742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x557615197faa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x557615233774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x5576151df742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x557615197faa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xa8449f (0x7fa89749549f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fa8974936cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fa8974969c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fa897498206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fa88d43eb8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fa8974967b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4908e23 (0x7fa88d437e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fa88d4389b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fa88d432af7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x49387d2 (0x7fa88d4677d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fa87eea4c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7fa8aeb7fbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7fa8cf1d36db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7fa8ceefc61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-12-01T11:13:32.7113246Z Traceback (most recent call last):
2022-12-01T11:13:32.7113596Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-12-01T11:13:32.7113827Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-12-01T11:13:32.7114458Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5911, in _gpu_add_wrong_gpus
2022-12-01T11:13:32.7114858Z     return x.cpu() + y.cuda()
2022-12-01T11:13:32.7115241Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-12-01T11:13:32.7115797Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-12-01T11:13:32.7116649Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa87eeb683b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7117631Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fa87eeb220e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7118552Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7fa889a1f0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7119350Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fa889a205af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7120261Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fa889a21db2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7121170Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fa889d0b53e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7121901Z frame #6: <unknown function> + 0x297c70e (0x7fa881a7570e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7122550Z frame #7: <unknown function> + 0x297c816 (0x7fa881a75816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7123364Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fa88a7b1de8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7124115Z frame #9: <unknown function> + 0x34c2400 (0x7fa88bff1400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7124759Z frame #10: <unknown function> + 0x34c2b79 (0x7fa88bff1b79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7125580Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fa88a7ea0e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7126316Z frame #12: <unknown function> + 0x2f75a4 (0x7fa896d085a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7127019Z frame #13: <unknown function> + 0x2f7996 (0x7fa896d08996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7127513Z frame #14: <unknown function> + 0x1ddc68 (0x55761522dc68 in /opt/conda/bin/python)
2022-12-01T11:13:32.7127983Z frame #15: <unknown function> + 0x199499 (0x5576151e9499 in /opt/conda/bin/python)
2022-12-01T11:13:32.7128385Z frame #16: <unknown function> + 0x1995fa (0x5576151e95fa in /opt/conda/bin/python)
2022-12-01T11:13:32.7128766Z frame #17: PyNumber_Add + 0x41 (0x5576151954b1 in /opt/conda/bin/python)
2022-12-01T11:13:32.7129181Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x557615232098 in /opt/conda/bin/python)
2022-12-01T11:13:32.7129599Z frame #19: <unknown function> + 0x18f742 (0x5576151df742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7129985Z frame #20: _PyObject_Call + 0x20a (0x557615197faa in /opt/conda/bin/python)
2022-12-01T11:13:32.7130368Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x557615233774 in /opt/conda/bin/python)
2022-12-01T11:13:32.7130771Z frame #22: <unknown function> + 0x18f742 (0x5576151df742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7131153Z frame #23: _PyObject_Call + 0x20a (0x557615197faa in /opt/conda/bin/python)
2022-12-01T11:13:32.7131754Z frame #24: <unknown function> + 0xa8449f (0x7fa89749549f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7132519Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fa8974936cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7133515Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fa8974969c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7134823Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fa897498206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7136019Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fa88d43eb8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7137317Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fa8974967b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7138224Z frame #30: <unknown function> + 0x4908e23 (0x7fa88d437e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7139180Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fa88d4389b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7140291Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fa88d432af7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7141108Z frame #33: <unknown function> + 0x49387d2 (0x7fa88d4677d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7141814Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fa87eea4c6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7142343Z frame #35: <unknown function> + 0xdbbf4 (0x7fa8aeb7fbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-12-01T11:13:32.7142972Z frame #36: <unknown function> + 0x76db (0x7fa8cf1d36db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-12-01T11:13:32.7143503Z frame #37: clone + 0x3f (0x7fa8ceefc61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-12-01T11:13:32.7143808Z 
2022-12-01T11:13:32.7143827Z 
2022-12-01T11:13:32.7143962Z On WorkerInfo(id=2, name=worker2):
2022-12-01T11:13:32.7156157Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f9955a2183b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f9955a1d20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7f996058a0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f996058b5af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f996058cdb2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f996087653e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x297c70e (0x7f99585e070e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #7: <unknown function> + 0x297c816 (0x7f99585e0816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f996131cde8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x34c2400 (0x7f9962b5c400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x34c2b79 (0x7f9962b5cb79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f99613550e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2f75a4 (0x7f996d8735a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2f7996 (0x7f996d873996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55dbcde85c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55dbcde41499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55dbcde415fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55dbcdded4b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55dbcde8a098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55dbcde37742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55dbcddeffaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55dbcde8b774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55dbcde37742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55dbcddeffaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xa8449f (0x7f996e00049f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f996dffe6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f996e0019c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f996e003206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f9963fa9b8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f996e0017b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4908e23 (0x7f9963fa2e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f9963fa39b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f9963f9daf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x49387d2 (0x7f9963fd27d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f9955a0fc6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7f99856eabf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f99a5d3e6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f99a5a6761f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-12-01T11:13:32.7163519Z Traceback (most recent call last):
2022-12-01T11:13:32.7164050Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 206, in _run_function
2022-12-01T11:13:32.7164516Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-12-01T11:13:32.7165099Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5911, in _gpu_add_wrong_gpus
2022-12-01T11:13:32.7165498Z     return x.cpu() + y.cuda()
2022-12-01T11:13:32.7165894Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-12-01T11:13:32.7166437Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-12-01T11:13:32.7167291Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f9955a2183b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7168243Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f9955a1d20e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7169144Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xc83 (0x7f996058a0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7169993Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f996058b5af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7170890Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f996058cdb2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7171834Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f996087653e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7172544Z frame #6: <unknown function> + 0x297c70e (0x7f99585e070e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7173186Z frame #7: <unknown function> + 0x297c816 (0x7f99585e0816 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so)
2022-12-01T11:13:32.7174009Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f996131cde8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7174941Z frame #9: <unknown function> + 0x34c2400 (0x7f9962b5c400 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7175571Z frame #10: <unknown function> + 0x34c2b79 (0x7f9962b5cb79 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7176314Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f99613550e2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7177024Z frame #12: <unknown function> + 0x2f75a4 (0x7f996d8735a4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7177677Z frame #13: <unknown function> + 0x2f7996 (0x7f996d873996 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7178148Z frame #14: <unknown function> + 0x1ddc68 (0x55dbcde85c68 in /opt/conda/bin/python)
2022-12-01T11:13:32.7178538Z frame #15: <unknown function> + 0x199499 (0x55dbcde41499 in /opt/conda/bin/python)
2022-12-01T11:13:32.7178937Z frame #16: <unknown function> + 0x1995fa (0x55dbcde415fa in /opt/conda/bin/python)
2022-12-01T11:13:32.7179386Z frame #17: PyNumber_Add + 0x41 (0x55dbcdded4b1 in /opt/conda/bin/python)
2022-12-01T11:13:32.7179797Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55dbcde8a098 in /opt/conda/bin/python)
2022-12-01T11:13:32.7180217Z frame #19: <unknown function> + 0x18f742 (0x55dbcde37742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7180622Z frame #20: _PyObject_Call + 0x20a (0x55dbcddeffaa in /opt/conda/bin/python)
2022-12-01T11:13:32.7181045Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55dbcde8b774 in /opt/conda/bin/python)
2022-12-01T11:13:32.7181438Z frame #22: <unknown function> + 0x18f742 (0x55dbcde37742 in /opt/conda/bin/python)
2022-12-01T11:13:32.7181836Z frame #23: _PyObject_Call + 0x20a (0x55dbcddeffaa in /opt/conda/bin/python)
2022-12-01T11:13:32.7182492Z frame #24: <unknown function> + 0xa8449f (0x7f996e00049f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7183225Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f996dffe6cd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7184234Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f996e0019c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7185338Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f996e003206 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7186624Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f9963fa9b8c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7187949Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f996e0017b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-12-01T11:13:32.7188830Z frame #30: <unknown function> + 0x4908e23 (0x7f9963fa2e23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7189762Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f9963fa39b8 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7190811Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f9963f9daf7 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7191605Z frame #33: <unknown function> + 0x49387d2 (0x7f9963fd27d2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-12-01T11:13:32.7192285Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f9955a0fc6b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-12-01T11:13:32.7192793Z frame #35: <unknown function> + 0xdbbf4 (0x7f99856eabf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-12-01T11:13:32.7193330Z frame #36: <unknown function> + 0x76db (0x7f99a5d3e6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-12-01T11:13:32.7193827Z frame #37: clone + 0x3f (0x7f99a5a6761f in /lib/x86_64-linux-gnu/libc.so.6)
2022-12-01T11:13:32.7194057Z 
2022-12-01T11:13:32.7194076Z 
2022-12-01T11:13:32.7194181Z ok (7.302s)
2022-12-01T11:13:32.7194328Z 
2022-12-01T11:13:32.7194600Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7194916Z Ran 1 test in 7.302s
2022-12-01T11:13:32.7195077Z 
2022-12-01T11:13:32.7195171Z OK
2022-12-01T11:13:32.7195307Z 
2022-12-01T11:13:32.7195432Z Generating XML reports...
2022-12-01T11:13:32.7196117Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110737.xml
2022-12-01T11:13:32.7196904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7197358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7197931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7198390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7198856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyqx76c3
2022-12-01T11:13:32.7199401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyqx76c3/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7199705Z 
2022-12-01T11:13:32.7199815Z Running tests...
2022-12-01T11:13:32.7200205Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7200780Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7201367Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7201925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60150
2022-12-01T11:13:32.7202336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60151
2022-12-01T11:13:32.7202861Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60152
2022-12-01T11:13:32.7203292Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60153
2022-12-01T11:13:32.7203934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7204431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7205009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7205423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7206007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7206451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7207070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7207518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7208098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7208540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7209109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7209550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7210128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7210573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7211125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7211598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7212064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcxp0b_85
2022-12-01T11:13:32.7212610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcxp0b_85/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7213124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporn04wib
2022-12-01T11:13:32.7213659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporn04wib/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7214188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkxwafbe
2022-12-01T11:13:32.7214907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkxwafbe/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7215442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfi_yke1
2022-12-01T11:13:32.7215968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfi_yke1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7216478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7216947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7217377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7217901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7218299Z fi_getinfo: -61
2022-12-01T11:13:32.7218558Z fi_getinfo: -61
2022-12-01T11:13:32.7218775Z fi_getinfo: -61
2022-12-01T11:13:32.7219045Z fi_getinfo: -61
2022-12-01T11:13:32.7219259Z ok (4.089s)
2022-12-01T11:13:32.7219404Z 
2022-12-01T11:13:32.7219673Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7219998Z Ran 1 test in 4.089s
2022-12-01T11:13:32.7220140Z 
2022-12-01T11:13:32.7220312Z OK
2022-12-01T11:13:32.7220454Z 
2022-12-01T11:13:32.7220579Z Generating XML reports...
2022-12-01T11:13:32.7221284Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110747.xml
2022-12-01T11:13:32.7222125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7222579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7223152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7223621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7224063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p_z3f61
2022-12-01T11:13:32.7224600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p_z3f61/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7224901Z 
2022-12-01T11:13:32.7225011Z Running tests...
2022-12-01T11:13:32.7225391Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7225955Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7226537Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7227064Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60337
2022-12-01T11:13:32.7227497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60338
2022-12-01T11:13:32.7227917Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60339
2022-12-01T11:13:32.7228342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60340
2022-12-01T11:13:32.7228957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7229394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7229967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7230440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7230997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7231444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7232014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7232475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7233035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7233488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7234057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7234506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7235076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7235519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7236083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7236521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7236988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cmmld7_
2022-12-01T11:13:32.7237586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cmmld7_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7238130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9f497vjh
2022-12-01T11:13:32.7238690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9f497vjh/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7239241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4xmpjjt
2022-12-01T11:13:32.7239779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4xmpjjt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7240310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb20guy_9
2022-12-01T11:13:32.7240849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb20guy_9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7241317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7241793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7242246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7242708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7243107Z fi_getinfo: -61
2022-12-01T11:13:32.7243364Z fi_getinfo: -61
2022-12-01T11:13:32.7243633Z fi_getinfo: -61
2022-12-01T11:13:32.7243898Z fi_getinfo: -61
2022-12-01T11:13:32.7244112Z ok (4.096s)
2022-12-01T11:13:32.7244257Z 
2022-12-01T11:13:32.7244529Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7244859Z Ran 1 test in 4.096s
2022-12-01T11:13:32.7245020Z 
2022-12-01T11:13:32.7245112Z OK
2022-12-01T11:13:32.7245225Z 
2022-12-01T11:13:32.7245352Z Generating XML reports...
2022-12-01T11:13:32.7246052Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110753.xml
2022-12-01T11:13:32.7246852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7247286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7247866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7248332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7248794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mqzz1mt
2022-12-01T11:13:32.7249311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mqzz1mt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7249610Z 
2022-12-01T11:13:32.7249719Z Running tests...
2022-12-01T11:13:32.7250118Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7250674Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7251283Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7251831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60524
2022-12-01T11:13:32.7252286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60525
2022-12-01T11:13:32.7252717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60526
2022-12-01T11:13:32.7253159Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60527
2022-12-01T11:13:32.7253768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7254219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7254976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7255500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7256081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7256582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7257153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7257619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7258191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7258613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7259180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7259640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7260199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7260646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7261213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7261674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7262118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxdi9xfa
2022-12-01T11:13:32.7262662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxdi9xfa/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7263199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelpsts4h
2022-12-01T11:13:32.7263740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelpsts4h/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7264266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpghwz89d1
2022-12-01T11:13:32.7264800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpghwz89d1/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7265335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdu2hs4d7
2022-12-01T11:13:32.7265867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdu2hs4d7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7266356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7266832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7267303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7267753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7268136Z fi_getinfo: -61
2022-12-01T11:13:32.7268376Z ok (10.122s)
2022-12-01T11:13:32.7268525Z 
2022-12-01T11:13:32.7268775Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7269106Z Ran 1 test in 10.123s
2022-12-01T11:13:32.7269272Z 
2022-12-01T11:13:32.7269362Z OK
2022-12-01T11:13:32.7269496Z 
2022-12-01T11:13:32.7269620Z Generating XML reports...
2022-12-01T11:13:32.7270307Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110759.xml
2022-12-01T11:13:32.7271103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7271556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7272131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7272584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7273105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph70ejs5c
2022-12-01T11:13:32.7273654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph70ejs5c/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7273999Z 
2022-12-01T11:13:32.7274092Z Running tests...
2022-12-01T11:13:32.7274498Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7275070Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7275673Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7276202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60739
2022-12-01T11:13:32.7276652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60740
2022-12-01T11:13:32.7277103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60741
2022-12-01T11:13:32.7277534Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60742
2022-12-01T11:13:32.7278136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7278593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7279170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7279619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7280200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7280646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7281198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7281668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7282248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7282697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7283302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7283715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7284289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7284735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7285281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7285751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7286215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c42itg4
2022-12-01T11:13:32.7286742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c42itg4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7287273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ewsr1ih
2022-12-01T11:13:32.7287809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ewsr1ih/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7288340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7agwquya
2022-12-01T11:13:32.7288856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7agwquya/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7289384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodqm13nf
2022-12-01T11:13:32.7289967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodqm13nf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7290485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7290940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7291453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7291921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7292300Z fi_getinfo: -61
2022-12-01T11:13:32.7292535Z ok (11.784s)
2022-12-01T11:13:32.7292684Z 
2022-12-01T11:13:32.7292954Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7293272Z Ran 1 test in 11.784s
2022-12-01T11:13:32.7293432Z 
2022-12-01T11:13:32.7293522Z OK
2022-12-01T11:13:32.7293654Z 
2022-12-01T11:13:32.7293778Z Generating XML reports...
2022-12-01T11:13:32.7294655Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110812.xml
2022-12-01T11:13:32.7295427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7295879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7296451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7296904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7297370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ikr8r4_
2022-12-01T11:13:32.7297909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ikr8r4_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7298205Z 
2022-12-01T11:13:32.7298315Z Running tests...
2022-12-01T11:13:32.7298701Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7299282Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7299885Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7300439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60955
2022-12-01T11:13:32.7300916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60956
2022-12-01T11:13:32.7301367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60957
2022-12-01T11:13:32.7301817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60958
2022-12-01T11:13:32.7302351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7302806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7303385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7303856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7304425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7304870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7305440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7305888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7306464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7306949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7307596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7308054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7308634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7309147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7309723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7310166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7310629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfaetcnl
2022-12-01T11:13:32.7311166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfaetcnl/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7311684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs0s1v83
2022-12-01T11:13:32.7312226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs0s1v83/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7312757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipu3hpa8
2022-12-01T11:13:32.7313299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipu3hpa8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7313810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17vwt4xk
2022-12-01T11:13:32.7314339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17vwt4xk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7314847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7315303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7315771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7316237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7316627Z fi_getinfo: -61
2022-12-01T11:13:32.7316846Z ok (11.898s)
2022-12-01T11:13:32.7316994Z 
2022-12-01T11:13:32.7317272Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7317602Z Ran 1 test in 11.898s
2022-12-01T11:13:32.7317765Z 
2022-12-01T11:13:32.7317839Z OK
2022-12-01T11:13:32.7317973Z 
2022-12-01T11:13:32.7318097Z Generating XML reports...
2022-12-01T11:13:32.7318800Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110826.xml
2022-12-01T11:13:32.7319593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7320022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7320596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7321068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7321517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgthf4a4y
2022-12-01T11:13:32.7322063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgthf4a4y/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7322366Z 
2022-12-01T11:13:32.7322475Z Running tests...
2022-12-01T11:13:32.7322884Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7323461Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7324068Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7324627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61171
2022-12-01T11:13:32.7325103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61172
2022-12-01T11:13:32.7325544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61173
2022-12-01T11:13:32.7325985Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61174
2022-12-01T11:13:32.7326688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7327129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7327704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7328176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7328758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7329186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7329762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7330234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7330798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7331246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7331811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7332280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7332843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7333294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7333870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7334543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7334999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhp5xdhx
2022-12-01T11:13:32.7335524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhp5xdhx/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7336065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ra0g9yf
2022-12-01T11:13:32.7336579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ra0g9yf/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7337108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw_auvjle
2022-12-01T11:13:32.7337640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw_auvjle/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7338168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpva0up2zg
2022-12-01T11:13:32.7338685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpva0up2zg/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7339188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7339667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7340133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7340584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7340970Z fi_getinfo: -61
2022-12-01T11:13:32.7341207Z ok (10.001s)
2022-12-01T11:13:32.7341335Z 
2022-12-01T11:13:32.7341604Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7341934Z Ran 1 test in 10.001s
2022-12-01T11:13:32.7342092Z 
2022-12-01T11:13:32.7342182Z OK
2022-12-01T11:13:32.7342312Z 
2022-12-01T11:13:32.7342417Z Generating XML reports...
2022-12-01T11:13:32.7343197Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110840.xml
2022-12-01T11:13:32.7344059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7344515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7345070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7345536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7346001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75y85l05
2022-12-01T11:13:32.7346535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75y85l05/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7346811Z 
2022-12-01T11:13:32.7346925Z Running tests...
2022-12-01T11:13:32.7347328Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7347905Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7348484Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7349025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61386
2022-12-01T11:13:32.7349479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61387
2022-12-01T11:13:32.7349928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61388
2022-12-01T11:13:32.7350352Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61389
2022-12-01T11:13:32.7350954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7351411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7351990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7352448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7353021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7353468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7354024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7354493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7355070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7355514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7356069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7356533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7357120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7357571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7358123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7358591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7359055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17otd3xd
2022-12-01T11:13:32.7359582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17otd3xd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7360172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklic0sf4
2022-12-01T11:13:32.7360716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklic0sf4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7361303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2q3qwnvs
2022-12-01T11:13:32.7361814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2q3qwnvs/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7362342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1q0kvpp
2022-12-01T11:13:32.7362884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1q0kvpp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7363402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7363856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7364331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7364799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7365171Z fi_getinfo: -61
2022-12-01T11:13:32.7365449Z fi_getinfo: -61
2022-12-01T11:13:32.7365721Z fi_getinfo: -61
2022-12-01T11:13:32.7365974Z fi_getinfo: -61
2022-12-01T11:13:32.7366204Z ok (17.220s)
2022-12-01T11:13:32.7366352Z 
2022-12-01T11:13:32.7366621Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7366941Z Ran 1 test in 17.220s
2022-12-01T11:13:32.7367101Z 
2022-12-01T11:13:32.7367194Z OK
2022-12-01T11:13:32.7367326Z 
2022-12-01T11:13:32.7367450Z Generating XML reports...
2022-12-01T11:13:32.7368155Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110852.xml
2022-12-01T11:13:32.7368930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7369384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7369961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7370437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7370884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0wzfp1wd
2022-12-01T11:13:32.7371423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0wzfp1wd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7371725Z 
2022-12-01T11:13:32.7371836Z Running tests...
2022-12-01T11:13:32.7372223Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7372799Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7373398Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7373934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61733
2022-12-01T11:13:32.7374535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61734
2022-12-01T11:13:32.7375021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61735
2022-12-01T11:13:32.7375429Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61736
2022-12-01T11:13:32.7376030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7376487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7377064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7377542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7378183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7378641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7379299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7379773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7380330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7380778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7381346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7381799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7382380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7382825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7383388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7383843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7384308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yjx5ah5
2022-12-01T11:13:32.7384847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yjx5ah5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7385377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohtw10_p
2022-12-01T11:13:32.7385901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohtw10_p/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7386429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02_mai3k
2022-12-01T11:13:32.7386956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02_mai3k/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7387458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19j36qff
2022-12-01T11:13:32.7387994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19j36qff/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7388492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7388967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7389420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7389883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7390270Z fi_getinfo: -61
2022-12-01T11:13:32.7390525Z fi_getinfo: -61
2022-12-01T11:13:32.7390789Z fi_getinfo: -61
2022-12-01T11:13:32.7391057Z fi_getinfo: -61
2022-12-01T11:13:32.7391274Z ok (19.827s)
2022-12-01T11:13:32.7391419Z 
2022-12-01T11:13:32.7391683Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7392019Z Ran 1 test in 19.827s
2022-12-01T11:13:32.7392175Z 
2022-12-01T11:13:32.7392267Z OK
2022-12-01T11:13:32.7392381Z 
2022-12-01T11:13:32.7392502Z Generating XML reports...
2022-12-01T11:13:32.7393204Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110911.xml
2022-12-01T11:13:32.7393998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7394430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7394817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7395057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7395328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy02n0lwk
2022-12-01T11:13:32.7395597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy02n0lwk/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7395662Z 
2022-12-01T11:13:32.7395777Z Running tests...
2022-12-01T11:13:32.7396044Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7396404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7396712Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7397477Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.575s)
2022-12-01T11:13:32.7397498Z 
2022-12-01T11:13:32.7397761Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7397860Z Ran 1 test in 1.575s
2022-12-01T11:13:32.7397879Z 
2022-12-01T11:13:32.7397985Z OK (skipped=1)
2022-12-01T11:13:32.7398006Z 
2022-12-01T11:13:32.7398130Z Generating XML reports...
2022-12-01T11:13:32.7398668Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110934.xml
2022-12-01T11:13:32.7399040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7399216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7399599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7399794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7400050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfov2b2cm
2022-12-01T11:13:32.7400307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfov2b2cm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7400326Z 
2022-12-01T11:13:32.7400434Z Running tests...
2022-12-01T11:13:32.7400696Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7401064Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7401376Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7401594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62120
2022-12-01T11:13:32.7401815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62121
2022-12-01T11:13:32.7402032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62122
2022-12-01T11:13:32.7402225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62123
2022-12-01T11:13:32.7402604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7402783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7403159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7403349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7403714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7403888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7404312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7404511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7404862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7405081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7405453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7405643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7406006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7406182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7406548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7406781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7407039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr54el_z
2022-12-01T11:13:32.7407295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr54el_z/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7407586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo84u3krd
2022-12-01T11:13:32.7407857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo84u3krd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7408107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd_lkcb4
2022-12-01T11:13:32.7408377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd_lkcb4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7408627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xy9x9ao
2022-12-01T11:13:32.7408895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xy9x9ao/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7409125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7409339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7409570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7409791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7409940Z fi_getinfo: -61
2022-12-01T11:13:32.7410077Z fi_getinfo: -61
2022-12-01T11:13:32.7410215Z fi_getinfo: -61
2022-12-01T11:13:32.7410349Z fi_getinfo: -61
2022-12-01T11:13:32.7410448Z ok (19.730s)
2022-12-01T11:13:32.7410468Z 
2022-12-01T11:13:32.7410716Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7410833Z Ran 1 test in 19.730s
2022-12-01T11:13:32.7410852Z 
2022-12-01T11:13:32.7410942Z OK
2022-12-01T11:13:32.7410961Z 
2022-12-01T11:13:32.7411090Z Generating XML reports...
2022-12-01T11:13:32.7411640Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110937.xml
2022-12-01T11:13:32.7412023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7412198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7412581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7412754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7413012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6p6ip2ka
2022-12-01T11:13:32.7413283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6p6ip2ka/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7413303Z 
2022-12-01T11:13:32.7413464Z Running tests...
2022-12-01T11:13:32.7413736Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7414100Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7414654Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7414881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62473
2022-12-01T11:13:32.7415100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62474
2022-12-01T11:13:32.7415298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62475
2022-12-01T11:13:32.7415509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62476
2022-12-01T11:13:32.7415898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7416078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7416464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7416658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7417034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7417210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7417584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7417756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7418122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7418300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7418669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7418862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7419229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7419402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7419771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7419941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7420197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppabtwy1o
2022-12-01T11:13:32.7420471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppabtwy1o/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7420727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyv8pbe2
2022-12-01T11:13:32.7420999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyv8pbe2/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7421257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuxf6eks
2022-12-01T11:13:32.7421527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuxf6eks/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7421777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tzq10qc
2022-12-01T11:13:32.7422037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tzq10qc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7422248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7422480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7422782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7423019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7423234Z fi_getinfo: -61
2022-12-01T11:13:32.7423371Z fi_getinfo: -61
2022-12-01T11:13:32.7423507Z fi_getinfo: -61
2022-12-01T11:13:32.7423625Z fi_getinfo: -61
2022-12-01T11:13:32.7423724Z ok (17.327s)
2022-12-01T11:13:32.7423743Z 
2022-12-01T11:13:32.7424012Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7424126Z Ran 1 test in 17.327s
2022-12-01T11:13:32.7424145Z 
2022-12-01T11:13:32.7424237Z OK
2022-12-01T11:13:32.7424256Z 
2022-12-01T11:13:32.7424381Z Generating XML reports...
2022-12-01T11:13:32.7424925Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110959.xml
2022-12-01T11:13:32.7425307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7425484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7425845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7426045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7426301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkd0a25p
2022-12-01T11:13:32.7426580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkd0a25p/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7426599Z 
2022-12-01T11:13:32.7426706Z Running tests...
2022-12-01T11:13:32.7426969Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7427329Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7427650Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7427850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62820
2022-12-01T11:13:32.7428071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62821
2022-12-01T11:13:32.7428288Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62822
2022-12-01T11:13:32.7428498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62823
2022-12-01T11:13:32.7428875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7429051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7429425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7429619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7429983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7430139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7430518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7430707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7431068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7431240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7431612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7431798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7432226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7432414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7432816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7433008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7433266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz99v8wof
2022-12-01T11:13:32.7433540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz99v8wof/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7433795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mwusku8
2022-12-01T11:13:32.7434063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mwusku8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7434321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rq50jd4
2022-12-01T11:13:32.7434585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rq50jd4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7434823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzkg27p5
2022-12-01T11:13:32.7435094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzkg27p5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7435319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7435551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7435779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7436006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7436151Z fi_getinfo: -61
2022-12-01T11:13:32.7436288Z fi_getinfo: -61
2022-12-01T11:13:32.7436406Z fi_getinfo: -61
2022-12-01T11:13:32.7436547Z fi_getinfo: -61
2022-12-01T11:13:32.7436644Z ok (15.802s)
2022-12-01T11:13:32.7436664Z 
2022-12-01T11:13:32.7436934Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7437049Z Ran 1 test in 15.802s
2022-12-01T11:13:32.7437069Z 
2022-12-01T11:13:32.7437158Z OK
2022-12-01T11:13:32.7437177Z 
2022-12-01T11:13:32.7437298Z Generating XML reports...
2022-12-01T11:13:32.7437849Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111019.xml
2022-12-01T11:13:32.7438201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7438381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7438766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7438958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7439211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngtsap17
2022-12-01T11:13:32.7439482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngtsap17/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7439502Z 
2022-12-01T11:13:32.7439610Z Running tests...
2022-12-01T11:13:32.7439877Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7440236Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7440535Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7440756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63166
2022-12-01T11:13:32.7440975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63167
2022-12-01T11:13:32.7441239Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63168
2022-12-01T11:13:32.7441457Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63169
2022-12-01T11:13:32.7441884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7442058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7442438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7442628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7442974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7443150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7443526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7443717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7444084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7444254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7444627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7444814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7445162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7445337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7445703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7445891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7446144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qruo3g_
2022-12-01T11:13:32.7446422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qruo3g_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7446673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptzy633mm
2022-12-01T11:13:32.7446943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptzy633mm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7447194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsj9zb76z
2022-12-01T11:13:32.7447441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsj9zb76z/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7447692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolg07hop
2022-12-01T11:13:32.7447957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolg07hop/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7448190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7448423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7448646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7448873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7449018Z fi_getinfo: -61
2022-12-01T11:13:32.7449137Z fi_getinfo: -61
2022-12-01T11:13:32.7449270Z fi_getinfo: -61
2022-12-01T11:13:32.7449407Z fi_getinfo: -61
2022-12-01T11:13:32.7449504Z ok (16.316s)
2022-12-01T11:13:32.7449524Z 
2022-12-01T11:13:32.7449785Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7449896Z Ran 1 test in 16.317s
2022-12-01T11:13:32.7449915Z 
2022-12-01T11:13:32.7450005Z OK
2022-12-01T11:13:32.7450076Z 
2022-12-01T11:13:32.7450208Z Generating XML reports...
2022-12-01T11:13:32.7450741Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111037.xml
2022-12-01T11:13:32.7451183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7451361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7451738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7451931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7452183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq906xvmy
2022-12-01T11:13:32.7452453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq906xvmy/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7452476Z 
2022-12-01T11:13:32.7452583Z Running tests...
2022-12-01T11:13:32.7452844Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7453189Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7453502Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7453721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63515
2022-12-01T11:13:32.7453934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63516
2022-12-01T11:13:32.7454145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63517
2022-12-01T11:13:32.7454539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63518
2022-12-01T11:13:32.7454916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7455114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7455438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7455635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7456000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7456170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7456545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7456739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7457103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7457277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7457649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7457823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7458192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7458369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7458743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7458928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7459184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpir70dkbj
2022-12-01T11:13:32.7459528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpir70dkbj/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7459793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e5szv5w
2022-12-01T11:13:32.7460119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e5szv5w/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7460356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7n1dwmy
2022-12-01T11:13:32.7460622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7n1dwmy/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7460874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0aez_7gr
2022-12-01T11:13:32.7461136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0aez_7gr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7461360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7461593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7461821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7462047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7462183Z fi_getinfo: -61
2022-12-01T11:13:32.7462322Z fi_getinfo: -61
2022-12-01T11:13:32.7462456Z fi_getinfo: -61
2022-12-01T11:13:32.7462587Z fi_getinfo: -61
2022-12-01T11:13:32.7462684Z ok (16.212s)
2022-12-01T11:13:32.7462703Z 
2022-12-01T11:13:32.7462970Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7463082Z Ran 1 test in 16.212s
2022-12-01T11:13:32.7463101Z 
2022-12-01T11:13:32.7463175Z OK
2022-12-01T11:13:32.7463210Z 
2022-12-01T11:13:32.7463315Z Generating XML reports...
2022-12-01T11:13:32.7463862Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111055.xml
2022-12-01T11:13:32.7464239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7464416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7464804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7464992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7465253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5rs0pcs
2022-12-01T11:13:32.7465523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5rs0pcs/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7465543Z 
2022-12-01T11:13:32.7465633Z Running tests...
2022-12-01T11:13:32.7465898Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7466258Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7466583Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7466801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63864
2022-12-01T11:13:32.7467021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63865
2022-12-01T11:13:32.7467234Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63866
2022-12-01T11:13:32.7467448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63867
2022-12-01T11:13:32.7467826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7467985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7468365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7468616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7468996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7469222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7469605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7469796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7470157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7470327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7470681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7470869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7471240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7471411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7471783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7471966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7472228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9qdetjr
2022-12-01T11:13:32.7472497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9qdetjr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7472735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpava2e0b5
2022-12-01T11:13:32.7473006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpava2e0b5/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7473261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7z47fsm
2022-12-01T11:13:32.7473530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7z47fsm/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7473783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbc9wk8_4
2022-12-01T11:13:32.7474049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbc9wk8_4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7474278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7474541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7474767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7474974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7475121Z fi_getinfo: -61
2022-12-01T11:13:32.7475257Z fi_getinfo: -61
2022-12-01T11:13:32.7475398Z fi_getinfo: -61
2022-12-01T11:13:32.7475533Z fi_getinfo: -61
2022-12-01T11:13:32.7475635Z ok (15.820s)
2022-12-01T11:13:32.7475655Z 
2022-12-01T11:13:32.7475922Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7476018Z Ran 1 test in 15.820s
2022-12-01T11:13:32.7476054Z 
2022-12-01T11:13:32.7476129Z OK
2022-12-01T11:13:32.7476148Z 
2022-12-01T11:13:32.7476273Z Generating XML reports...
2022-12-01T11:13:32.7476820Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111114.xml
2022-12-01T11:13:32.7477194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7477371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7477799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7478004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7478256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mryo725
2022-12-01T11:13:32.7478553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mryo725/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7478590Z 
2022-12-01T11:13:32.7478681Z Running tests...
2022-12-01T11:13:32.7478947Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7479306Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7479670Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7479888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64210
2022-12-01T11:13:32.7480107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64211
2022-12-01T11:13:32.7480320Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64212
2022-12-01T11:13:32.7480529Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64213
2022-12-01T11:13:32.7480891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7481065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7481440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7481630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7481994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7482166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7482540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7482731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7483078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7483251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7483617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7483808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7484163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7484335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7484714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7484901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7485160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rsyc450
2022-12-01T11:13:32.7485412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rsyc450/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7485666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp346ush68
2022-12-01T11:13:32.7485918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_26bjav
2022-12-01T11:13:32.7486184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp346ush68/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7486439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_26bjav/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7486692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j1j0ne8
2022-12-01T11:13:32.7487003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j1j0ne8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7487270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7487533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7487759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7487981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7488128Z fi_getinfo: -61
2022-12-01T11:13:32.7488263Z fi_getinfo: -61
2022-12-01T11:13:32.7488396Z fi_getinfo: -61
2022-12-01T11:13:32.7488528Z fi_getinfo: -61
2022-12-01T11:13:32.7488609Z ok (17.437s)
2022-12-01T11:13:32.7488645Z 
2022-12-01T11:13:32.7488894Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7489007Z Ran 1 test in 17.437s
2022-12-01T11:13:32.7489030Z 
2022-12-01T11:13:32.7489120Z OK
2022-12-01T11:13:32.7489140Z 
2022-12-01T11:13:32.7489262Z Generating XML reports...
2022-12-01T11:13:32.7489809Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111132.xml
2022-12-01T11:13:32.7490183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7490357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7490734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7490907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7491158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5il8otxx
2022-12-01T11:13:32.7491428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5il8otxx/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7491448Z 
2022-12-01T11:13:32.7491556Z Running tests...
2022-12-01T11:13:32.7491817Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7492179Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7492526Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7492710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64557
2022-12-01T11:13:32.7492970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64558
2022-12-01T11:13:32.7493124Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64559
2022-12-01T11:13:32.7493334Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64560
2022-12-01T11:13:32.7493708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7493885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7494271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7494645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7495024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7495160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7495516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7495710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7496142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7496324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7496696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7496943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7497314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7497486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7497857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7498027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7498280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp556_8cot
2022-12-01T11:13:32.7498548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp556_8cot/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7498799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02kz9nde
2022-12-01T11:13:32.7499068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02kz9nde/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7499322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv61_0qga
2022-12-01T11:13:32.7499582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv61_0qga/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7499861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl9vwvavt
2022-12-01T11:13:32.7500103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl9vwvavt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7500335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7500556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7500822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7500949Z fi_getinfo: -61
2022-12-01T11:13:32.7501191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7501333Z fi_getinfo: -61
2022-12-01T11:13:32.7501466Z fi_getinfo: -61
2022-12-01T11:13:32.7501587Z fi_getinfo: -61
2022-12-01T11:13:32.7501679Z ok (19.619s)
2022-12-01T11:13:32.7501699Z 
2022-12-01T11:13:32.7501959Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7502022Z Ran 1 test in 19.619s
2022-12-01T11:13:32.7502042Z 
2022-12-01T11:13:32.7502134Z OK
2022-12-01T11:13:32.7502157Z 
2022-12-01T11:13:32.7502277Z Generating XML reports...
2022-12-01T11:13:32.7502821Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111152.xml
2022-12-01T11:13:32.7503195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7503354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7503739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7503927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7504185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmg1pdb4
2022-12-01T11:13:32.7504453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmg1pdb4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7504473Z 
2022-12-01T11:13:32.7504579Z Running tests...
2022-12-01T11:13:32.7504839Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7505291Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7505617Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7505863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64910
2022-12-01T11:13:32.7506080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64911
2022-12-01T11:13:32.7506293Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64912
2022-12-01T11:13:32.7506504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64913
2022-12-01T11:13:32.7506919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7507095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7507474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7507667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7508014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7508190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7508561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7508766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7509107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7509278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7509642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7509831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7510196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7510381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7510756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7510937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7511238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27bshys0
2022-12-01T11:13:32.7511485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27bshys0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7511717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1k36ldyg
2022-12-01T11:13:32.7511984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1k36ldyg/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7512238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8h1asw7g
2022-12-01T11:13:32.7512488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8h1asw7g/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7512744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcop21j8
2022-12-01T11:13:32.7513007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcop21j8/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7513234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7513461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7513686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7513906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7514051Z fi_getinfo: -61
2022-12-01T11:13:32.7514234Z fi_getinfo: -61
2022-12-01T11:13:32.7514365Z fi_getinfo: -61
2022-12-01T11:13:32.7514499Z fi_getinfo: -61
2022-12-01T11:13:32.7514597Z ok (17.221s)
2022-12-01T11:13:32.7514671Z 
2022-12-01T11:13:32.7514943Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7515058Z Ran 1 test in 17.221s
2022-12-01T11:13:32.7515078Z 
2022-12-01T11:13:32.7515169Z OK
2022-12-01T11:13:32.7515189Z 
2022-12-01T11:13:32.7515310Z Generating XML reports...
2022-12-01T11:13:32.7515841Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111213.xml
2022-12-01T11:13:32.7516212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7516386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7516772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7516964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7517221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxktovwrb
2022-12-01T11:13:32.7517493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxktovwrb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7517513Z 
2022-12-01T11:13:32.7517621Z Running tests...
2022-12-01T11:13:32.7517883Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7518246Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7518537Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7518754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65257
2022-12-01T11:13:32.7518973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65258
2022-12-01T11:13:32.7519186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65259
2022-12-01T11:13:32.7519403Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65260
2022-12-01T11:13:32.7519775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7519950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7520314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7520503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7520867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7521042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7521422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7521610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7521978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7522151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7522521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7522691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7523051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7523222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7523646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7523842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7524141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqfc7l4h
2022-12-01T11:13:32.7524410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqfc7l4h/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7524662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi02bv5aw
2022-12-01T11:13:32.7524910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi02bv5aw/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7525162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdiryynb
2022-12-01T11:13:32.7525427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdiryynb/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7525681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzq0duypa
2022-12-01T11:13:32.7525945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzq0duypa/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7526175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7526394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7526619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7526841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7526973Z fi_getinfo: -61
2022-12-01T11:13:32.7527112Z fi_getinfo: -61
2022-12-01T11:13:32.7527246Z fi_getinfo: -61
2022-12-01T11:13:32.7527381Z fi_getinfo: -61
2022-12-01T11:13:32.7527478Z ok (19.732s)
2022-12-01T11:13:32.7527498Z 
2022-12-01T11:13:32.7527762Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7527877Z Ran 1 test in 19.732s
2022-12-01T11:13:32.7527897Z 
2022-12-01T11:13:32.7527970Z OK
2022-12-01T11:13:32.7527989Z 
2022-12-01T11:13:32.7528112Z Generating XML reports...
2022-12-01T11:13:32.7528666Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111233.xml
2022-12-01T11:13:32.7529039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7529215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7529595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7529788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7530042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgnfwojc
2022-12-01T11:13:32.7530314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgnfwojc/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7530334Z 
2022-12-01T11:13:32.7530425Z Running tests...
2022-12-01T11:13:32.7530698Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7531060Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7531378Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7531595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65610
2022-12-01T11:13:32.7531810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65611
2022-12-01T11:13:32.7532022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65612
2022-12-01T11:13:32.7532232Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65613
2022-12-01T11:13:32.7532638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7532824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7533260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7533450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7533812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7533983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7534558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7534745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7535123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7535238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7535611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7535796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7536155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7536325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7536700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7536889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7537145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoxzwq6bi
2022-12-01T11:13:32.7537384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ol0d8ni
2022-12-01T11:13:32.7537656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoxzwq6bi/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7537926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ol0d8ni/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7538179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjpitcfvu
2022-12-01T11:13:32.7538447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjpitcfvu/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7538698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zf38g9m
2022-12-01T11:13:32.7538959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zf38g9m/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7539188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7539419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7539629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7539858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7540004Z fi_getinfo: -61
2022-12-01T11:13:32.7540142Z fi_getinfo: -61
2022-12-01T11:13:32.7540275Z fi_getinfo: -61
2022-12-01T11:13:32.7540410Z fi_getinfo: -61
2022-12-01T11:13:32.7540509Z ok (7.305s)
2022-12-01T11:13:32.7540529Z 
2022-12-01T11:13:32.7540779Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7540890Z Ran 1 test in 7.305s
2022-12-01T11:13:32.7540909Z 
2022-12-01T11:13:32.7540999Z OK
2022-12-01T11:13:32.7541018Z 
2022-12-01T11:13:32.7541140Z Generating XML reports...
2022-12-01T11:13:32.7541760Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111255.xml
2022-12-01T11:13:32.7542146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7542385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7542768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7542960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7543193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45b9zwzx
2022-12-01T11:13:32.7543456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45b9zwzx/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7543477Z 
2022-12-01T11:13:32.7543583Z Running tests...
2022-12-01T11:13:32.7543845Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7544206Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7544513Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7544732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65961
2022-12-01T11:13:32.7544948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65962
2022-12-01T11:13:32.7545146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65963
2022-12-01T11:13:32.7545357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65964
2022-12-01T11:13:32.7545730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7545907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7546288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7546478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7546840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7547014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7547388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7547560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7547918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7548089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7548455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7548644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7549008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7549181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7549549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7549719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7549972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xe2wspg
2022-12-01T11:13:32.7550238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xe2wspg/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7550493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjvzibd_
2022-12-01T11:13:32.7550816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjvzibd_/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7551075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmmhswzbt
2022-12-01T11:13:32.7551387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmmhswzbt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7551635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdneu6yy4
2022-12-01T11:13:32.7551901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdneu6yy4/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7552111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7552337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7552565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7552789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7552937Z fi_getinfo: -61
2022-12-01T11:13:32.7553072Z fi_getinfo: -61
2022-12-01T11:13:32.7553211Z fi_getinfo: -61
2022-12-01T11:13:32.7553328Z fi_getinfo: -61
2022-12-01T11:13:32.7553430Z ok (8.903s)
2022-12-01T11:13:32.7553450Z 
2022-12-01T11:13:32.7553714Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7553826Z Ran 1 test in 8.904s
2022-12-01T11:13:32.7553846Z 
2022-12-01T11:13:32.7553936Z OK
2022-12-01T11:13:32.7553956Z 
2022-12-01T11:13:32.7554076Z Generating XML reports...
2022-12-01T11:13:32.7554623Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111304.xml
2022-12-01T11:13:32.7554994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7555167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7555534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7555726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7555985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpschueikd
2022-12-01T11:13:32.7556258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpschueikd/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7556278Z 
2022-12-01T11:13:32.7556386Z Running tests...
2022-12-01T11:13:32.7556649Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7557006Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7557328Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7557529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66608
2022-12-01T11:13:32.7557748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66609
2022-12-01T11:13:32.7557962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66610
2022-12-01T11:13:32.7558176Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66611
2022-12-01T11:13:32.7558551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7558725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7559105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7559295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7559656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7559860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7560245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7560480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7560842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7561011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7561383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7561568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7561932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7562086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7562456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7562640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7562897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0di3w3u
2022-12-01T11:13:32.7563165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0di3w3u/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7563419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphk38wdda
2022-12-01T11:13:32.7563686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphk38wdda/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7563937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppoaey3e0
2022-12-01T11:13:32.7564201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppoaey3e0/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7564437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8cdxcrt
2022-12-01T11:13:32.7564699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8cdxcrt/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7564928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7565155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7565381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7565605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7565752Z skip: Need at least 4 CUDA devices (3.890s)
2022-12-01T11:13:32.7565772Z 
2022-12-01T11:13:32.7566041Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7566135Z Ran 1 test in 3.890s
2022-12-01T11:13:32.7566171Z 
2022-12-01T11:13:32.7566261Z OK (skipped=1)
2022-12-01T11:13:32.7566284Z 
2022-12-01T11:13:32.7566407Z Generating XML reports...
2022-12-01T11:13:32.7566972Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201111315.xml
2022-12-01T11:13:32.7567347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7567520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7567897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7568089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7568343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaf9lgc85
2022-12-01T11:13:32.7568594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaf9lgc85/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7568679Z 
2022-12-01T11:13:32.7568777Z Running tests...
2022-12-01T11:13:32.7569041Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7569447Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7569767Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7569987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66779
2022-12-01T11:13:32.7570205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66780
2022-12-01T11:13:32.7570419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66781
2022-12-01T11:13:32.7570627Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66782
2022-12-01T11:13:32.7570988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7571167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7571549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7571740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7572101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7572273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7572641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7572831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7573179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7573356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7573732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7573921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7574281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7574648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7575032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7575183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7575441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswewl7pr
2022-12-01T11:13:32.7575699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswewl7pr/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7575951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga_a90h7
2022-12-01T11:13:32.7576218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga_a90h7/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7576468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3u07la4t
2022-12-01T11:13:32.7576731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3u07la4t/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7576979Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_y1v73q9
2022-12-01T11:13:32.7577237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_y1v73q9/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7577464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7577675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7577972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7578207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7578429Z skip: Need at least 4 CUDA devices (3.794s)
2022-12-01T11:13:32.7578450Z 
2022-12-01T11:13:32.7578718Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7578829Z Ran 1 test in 3.794s
2022-12-01T11:13:32.7578849Z 
2022-12-01T11:13:32.7578956Z OK (skipped=1)
2022-12-01T11:13:32.7578975Z 
2022-12-01T11:13:32.7579099Z Generating XML reports...
2022-12-01T11:13:32.7579664Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201111322.xml
2022-12-01T11:13:32.7580016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7580195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7580577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7580771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7581026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4rr773h
2022-12-01T11:13:32.7581295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4rr773h/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7581315Z 
2022-12-01T11:13:32.7581422Z Running tests...
2022-12-01T11:13:32.7581686Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7582044Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-12-01T11:13:32.7582351Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:13:32.7582573Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66950
2022-12-01T11:13:32.7582791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66951
2022-12-01T11:13:32.7583004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66952
2022-12-01T11:13:32.7583213Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66953
2022-12-01T11:13:32.7583586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7583759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7584137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7584310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7584675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7584847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7585222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7585415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7585772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7585945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7586314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7586500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7586848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:13:32.7587068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:13:32.7587446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:13:32.7587679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:13:32.7587938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmq03r23r
2022-12-01T11:13:32.7588208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmq03r23r/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7588461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhjxstjx
2022-12-01T11:13:32.7588729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhjxstjx/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7588963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9uu1wzj
2022-12-01T11:13:32.7589233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9uu1wzj/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7589482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6ta7ahp
2022-12-01T11:13:32.7589748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6ta7ahp/_remote_module_non_scriptable.py
2022-12-01T11:13:32.7589976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:13:32.7590202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:13:32.7590428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:13:32.7590651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:13:32.7590800Z skip: Need at least 4 CUDA devices (3.867s)
2022-12-01T11:13:32.7590820Z 
2022-12-01T11:13:32.7591075Z ----------------------------------------------------------------------
2022-12-01T11:13:32.7591189Z Ran 1 test in 3.867s
2022-12-01T11:13:32.7591209Z 
2022-12-01T11:13:32.7591316Z OK (skipped=1)
2022-12-01T11:13:32.7591336Z 
2022-12-01T11:13:32.7591458Z Generating XML reports...
2022-12-01T11:13:32.7592028Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201111328.xml
2022-12-01T11:13:32.7592047Z 
2022-12-01T11:13:32.7592426Z ##[endgroup]
2022-12-01T11:13:32.7592940Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_jxydwv5c)
2022-12-01T11:13:32.7592961Z 
2022-12-01T11:13:32.7593256Z Running distributed/optim/test_zero_redundancy_optimizer ... [2022-12-01 11:13:32.487956]
2022-12-01T11:13:32.7593744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:13:32.488362]
2022-12-01T11:16:12.1854072Z 
2022-12-01T11:16:12.1855031Z Expand the folded group to see the log file of distributed/optim/test_zero_redundancy_optimizer
2022-12-01T11:16:12.1856060Z ##[group]PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_ijiotkcl)
2022-12-01T11:16:12.1858380Z 
2022-12-01T11:16:12.1860783Z Running tests...
2022-12-01T11:16:12.1861382Z ----------------------------------------------------------------------
2022-12-01T11:16:12.1861957Z Test results will be stored in test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer
2022-12-01T11:16:12.1862526Z   test_add_param_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1863028Z Check that ZeroRedundancyOptimizer properly handles adding a new ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:16:12.1864503Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67287 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.594s)
2022-12-01T11:16:12.1865552Z   test_collect_shards (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1866327Z Check the state consolidation mechanism and the state dict exposed ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67121
2022-12-01T11:16:12.1867024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67122
2022-12-01T11:16:12.1867762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1868193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1868828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1869254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1869873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1870345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1870935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1871364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1871778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1872253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1872765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1873273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1873948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1874622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1875288Z ok (5.216s)
2022-12-01T11:16:12.1875788Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1876507Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67205
2022-12-01T11:16:12.1877040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67206
2022-12-01T11:16:12.1877695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1878137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1878721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1879211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1879780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1880216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1880792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1881258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1881726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1882281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1882897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1883568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1884225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1884907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1885849Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1886902Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1887579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1888548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1889384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1890217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1890651Z ok (4.312s)
2022-12-01T11:16:12.1891237Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1892023Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67318
2022-12-01T11:16:12.1892710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67319
2022-12-01T11:16:12.1893408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1893868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1895163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1895684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1896311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1896716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1897367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1897771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1898198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1898700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1899193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1899733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1900323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1901342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1902389Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1903469Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1904208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1904701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1905187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1905749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1906009Z ok (4.212s)
2022-12-01T11:16:12.1906526Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1907239Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67431
2022-12-01T11:16:12.1907774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67432
2022-12-01T11:16:12.1908402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1908860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1909443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1909899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1910532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1910927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1911575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1911959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1912407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1912897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1913357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1913853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1914513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1915272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1916095Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1917153Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1917895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1918318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1918781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1919259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1919673Z ok (4.314s)
2022-12-01T11:16:12.1920227Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1920960Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67544
2022-12-01T11:16:12.1921512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67545
2022-12-01T11:16:12.1922134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1922593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1923160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1923628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1924214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1924644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1925216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1925736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1926181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1926654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1927142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1927636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1928283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1928976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1929884Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1930927Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1931592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1932065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1932551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1933034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1933368Z ok (4.212s)
2022-12-01T11:16:12.1933876Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1935499Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67657
2022-12-01T11:16:12.1936067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67658
2022-12-01T11:16:12.1936674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1937131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1937816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1938345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1939037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1939516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1940130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1940618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1941085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1941603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1942127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1942628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1943328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1944059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1945016Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1946094Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1946819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1947330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1947845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1948335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1948705Z ok (4.212s)
2022-12-01T11:16:12.1949237Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1949987Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67770
2022-12-01T11:16:12.1950554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67771
2022-12-01T11:16:12.1951202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1951689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1952284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1952781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1953393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1953870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1954460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1954957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1955491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1956024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1956593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1957117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1957821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1958535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1959492Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1960603Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1961317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1961832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1962325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1962837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1963208Z ok (4.232s)
2022-12-01T11:16:12.1963722Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1964472Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67883
2022-12-01T11:16:12.1965054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67884
2022-12-01T11:16:12.1965712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1966174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1966784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1967292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1967908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1968367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1968978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1969477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1969922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1970445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1970972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1971473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1972152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1972888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1973930Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1975621Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1976280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1976779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1977266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1977748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1978083Z ok (4.212s)
2022-12-01T11:16:12.1978595Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1979303Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67996
2022-12-01T11:16:12.1979859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67997
2022-12-01T11:16:12.1980463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1980919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1981507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1981962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1982549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1982999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1983573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1984029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1984467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1984955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1985422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1985905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.1986564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1987257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.1988150Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1989203Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.1989878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1990366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1990828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1991406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.1991776Z ok (4.212s)
2022-12-01T11:16:12.1992292Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.1993056Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68109
2022-12-01T11:16:12.1993607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68110
2022-12-01T11:16:12.1994224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1994680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1995237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1995716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1996298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.1996732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.1997311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.1997775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.1998218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.1998693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.1999179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.1999671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2000329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2001003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2001911Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2002956Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2003621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2004092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2004573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2005057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2005388Z ok (4.312s)
2022-12-01T11:16:12.2005898Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2006606Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68222
2022-12-01T11:16:12.2007152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68223
2022-12-01T11:16:12.2007750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2008301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2008897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2009436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2009999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2010448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2011026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2011500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2011922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2012424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2012914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2013376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2014039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2015615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2016539Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2017575Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2018248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2018741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2019228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2019689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2020041Z ok (4.212s)
2022-12-01T11:16:12.2020548Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2021253Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68335
2022-12-01T11:16:12.2021780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68336
2022-12-01T11:16:12.2022393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2022907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2023419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2023893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2024478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2024924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2025475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2025989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2026529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2027024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2027570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2028071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2028735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2029401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2030318Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2031365Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2032032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2032517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2032982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2033462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2033810Z ok (4.212s)
2022-12-01T11:16:12.2034295Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2035001Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68448
2022-12-01T11:16:12.2035556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68449
2022-12-01T11:16:12.2036176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2036610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2037191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2037669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2038293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2038674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2039354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2039721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2040150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2040644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2041130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2041692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2042255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2042949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2043922Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2045041Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2045692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2046181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2046660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2047159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2047524Z ok (4.212s)
2022-12-01T11:16:12.2047986Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2048707Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68561
2022-12-01T11:16:12.2049254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68562
2022-12-01T11:16:12.2049851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2050306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2050886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2051341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2051922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2052367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2052941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2053387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2053827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2055055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2055604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2056093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2056771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2057492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2058367Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2059495Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2060142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2060630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2061143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2061624Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2062053Z ok (4.212s)
2022-12-01T11:16:12.2062564Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2063255Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68674
2022-12-01T11:16:12.2063803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68675
2022-12-01T11:16:12.2064424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2064887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2065450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2065923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2066510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2066998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2067517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2067980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2068416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2068891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2069375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2069864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2070549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2071248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2072102Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2073144Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2073811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2074282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2074772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2075250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2075596Z ok (4.212s)
2022-12-01T11:16:12.2076080Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2076780Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68787
2022-12-01T11:16:12.2077325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68788
2022-12-01T11:16:12.2077992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2078463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2079044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2079581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2080147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2080599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2081177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2081648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2082068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2082546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2083039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2083522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2084186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2084948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2085796Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2086926Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2087512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2088052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2088464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2088927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2089285Z ok (4.212s)
2022-12-01T11:16:12.2089783Z   test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2090487Z Check that overlapping DDP with ZeRO using the given method determined ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68900
2022-12-01T11:16:12.2091019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68901
2022-12-01T11:16:12.2091674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2092098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2092735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2093129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2093711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2094154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2095230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2095777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2096299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2096810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2097281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2097829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2098496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2099182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2100079Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2101103Z INFO:torch.distributed.optim.zero_redundancy_optimizer:Using the functional optimizer <class 'torch.distributed.optim.functional_sgd._FunctionalSGD'> instead of <class 'torch.optim.sgd.SGD'> since `overlap_with_ddp=True`
2022-12-01T11:16:12.2101799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2102298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2102762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2103239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2103590Z ok (4.212s)
2022-12-01T11:16:12.2103974Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2104572Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69013
2022-12-01T11:16:12.2105117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69014
2022-12-01T11:16:12.2105733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2106170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2106750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2107224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2107803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2108298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2108803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2109269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2109739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2110171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2110651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2111178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2111836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2112599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2113157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8tfv5zd2
2022-12-01T11:16:12.2113701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8tfv5zd2/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2114271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ln4zvmo
2022-12-01T11:16:12.2114862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ln4zvmo/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2115887Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2117353Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2118485Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2119241Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2119997Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2120724Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2121460Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2122204Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2122943Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2123690Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2124489Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2125181Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2125949Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2126733Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2127434Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2128230Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2128970Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2129705Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2130175Z ok (4.615s)
2022-12-01T11:16:12.2130583Z   test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2131216Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69096
2022-12-01T11:16:12.2131885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69097
2022-12-01T11:16:12.2132362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2132822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2133418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2133872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2135364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2135770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2136362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2136839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2137266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2137760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2138255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2138722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2139404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2140068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2140671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsis4m561
2022-12-01T11:16:12.2141237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsis4m561/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2141723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj6tn1gp
2022-12-01T11:16:12.2142245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj6tn1gp/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2143380Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2144952Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2146115Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2146847Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2147597Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2148344Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2149136Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2149817Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2150527Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2151264Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2152046Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2152777Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2153594Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2154181Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2154915Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2155653Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2156384Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2157092Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2157560Z ok (4.615s)
2022-12-01T11:16:12.2157994Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2158683Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69179
2022-12-01T11:16:12.2159274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69180
2022-12-01T11:16:12.2159965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2160426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2160996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2161471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2162048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2162497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2163057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2163525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2163968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2164448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2164917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2165412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2166071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2166740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2167300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqr24v9k
2022-12-01T11:16:12.2167842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqr24v9k/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2168378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8h1q9br
2022-12-01T11:16:12.2168891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8h1q9br/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2169944Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2171450Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2172561Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2173311Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2174119Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2175757Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2176594Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2177331Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2178075Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2178816Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2179529Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2180260Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2180989Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2181726Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2182457Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2183170Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2183897Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2184627Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2185098Z ok (4.615s)
2022-12-01T11:16:12.2185507Z   test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2186137Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69262
2022-12-01T11:16:12.2186674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69263
2022-12-01T11:16:12.2187314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2187752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2188376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2188804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2189363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2189805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2190460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2190942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2191365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2191922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2192414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2192902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2193545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2194235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2194790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60kmpuhs
2022-12-01T11:16:12.2195313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60kmpuhs/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2195850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr4uwz63
2022-12-01T11:16:12.2196385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr4uwz63/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2197438Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2198943Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2200073Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2200828Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2201560Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2202302Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2203041Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2203775Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2204488Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2205297Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2206050Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2206842Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2207577Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2208288Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2209020Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2209753Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2210493Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2211218Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2211670Z ok (4.515s)
2022-12-01T11:16:12.2212098Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2212726Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69345
2022-12-01T11:16:12.2213249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69346
2022-12-01T11:16:12.2213871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2214663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2215174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2215596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2216179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2216628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2217203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2217658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2218100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2218579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2219050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2219548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2220208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2220901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2221432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2gtl7y_
2022-12-01T11:16:12.2222065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2gtl7y_/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2222614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_it491tk
2022-12-01T11:16:12.2223225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_it491tk/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2224258Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2225812Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2226949Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2227706Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2228501Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2229185Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2229901Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2230637Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2231374Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2232113Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2232845Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2233560Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2234288Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2235016Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2235809Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2236559Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2237326Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2238066Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2238540Z ok (4.515s)
2022-12-01T11:16:12.2238966Z   test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2239571Z When combined with DDP, check that a local optimizer gives the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69428
2022-12-01T11:16:12.2240113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69429
2022-12-01T11:16:12.2240737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2241179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2241770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2242240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2242822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2243247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2243820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2244290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2244733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2245209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2245700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2246195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2246834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2247529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2248079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz2ly7rr
2022-12-01T11:16:12.2248625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz2ly7rr/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2249136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstb4fbie
2022-12-01T11:16:12.2249677Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstb4fbie/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2250730Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2252305Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:16:12.2253534Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2254281Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2255177Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2255924Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2256670Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2257410Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2258143Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2258856Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2259590Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2260327Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2261058Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2261784Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2262494Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2263226Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2263961Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2264686Z WARNING:torch.distributed.optim.zero_redundancy_optimizer:ZeroRedundancyOptimizer detected that the trainable parameters changed; rebuilding the parameter buckets if enabled
2022-12-01T11:16:12.2265136Z ok (4.634s)
2022-12-01T11:16:12.2265492Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2266042Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69511
2022-12-01T11:16:12.2266569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69512
2022-12-01T11:16:12.2267265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2267791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2268375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2268835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2269417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2269865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2270446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2270895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2271337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2271831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2272307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2272793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2273457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2274146Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2274524Z ok (4.814s)
2022-12-01T11:16:12.2274898Z   test_multiple_param_groups (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2275490Z Check parity between constructing ZeRO with multiple parameter groups ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69595
2022-12-01T11:16:12.2276041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69596
2022-12-01T11:16:12.2276640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2277096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2277674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2278128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2278710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2279158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2279735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2280185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2280626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2281123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2281610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2282072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2282723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2283414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2283788Z ok (5.215s)
2022-12-01T11:16:12.2284232Z   test_nondefault_process_group (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2284986Z Check that ZeroRedundancyOptimizer works with a non-default process ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69679
2022-12-01T11:16:12.2285615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69680
2022-12-01T11:16:12.2286207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2286656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2287248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2287685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2288263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2288709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2289281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2289730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2290171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2290642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2291167Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-12-01T11:16:12.2291711Z INFO:torch.testing._internal.common_distributed:Skipping `test_nondefault_process_group()` since world size of 2 is less than 4
2022-12-01T11:16:12.2292088Z ok (2.110s)
2022-12-01T11:16:12.2292438Z   test_sharding (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2293583Z Check ZeroRedundancyOptimizer's parameter sharding at construction ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/67295 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-12-01T11:16:12.2294656Z   test_step (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2295161Z Check that ZeroRedundancyOptimizer properly exposes the ``step()`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69747
2022-12-01T11:16:12.2295708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69748
2022-12-01T11:16:12.2296329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2296763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2297348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2297820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2298386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2298835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2299407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2299875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2300296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2300787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2301432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2301922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2302661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2303354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2303752Z ok (4.113s)
2022-12-01T11:16:12.2304095Z   test_step_with_closure (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2304668Z Check that ZeroRedundancyOptimizer properly exposes the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69830
2022-12-01T11:16:12.2305209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69831
2022-12-01T11:16:12.2305823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2306260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2306842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2307320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2307885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2308330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2308901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2309366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2309786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2310282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2310772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2311241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2311900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2312588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2312987Z ok (4.213s)
2022-12-01T11:16:12.2313327Z   test_zero_join_cpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2313884Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69913
2022-12-01T11:16:12.2314423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69914
2022-12-01T11:16:12.2315035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2315472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2316047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2316515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2317077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2317521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2318090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2318554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2319053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2319544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2320089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2320567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2321236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2321928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2322481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6z0js0s
2022-12-01T11:16:12.2322994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg73z9qo
2022-12-01T11:16:12.2323532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6z0js0s/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2324080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg73z9qo/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2324598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2325068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2325780Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpq6z0js0s'>
2022-12-01T11:16:12.2326249Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:16:12.2326819Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpeg73z9qo'>
2022-12-01T11:16:12.2327274Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:16:12.2327560Z ok (2.208s)
2022-12-01T11:16:12.2327919Z   test_zero_join_gpu (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2328455Z Check that the ZeRO join hook allows training with uneven inputs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69991
2022-12-01T11:16:12.2328992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69992
2022-12-01T11:16:12.2329602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2330035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2330617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2331088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2331669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2332098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2332672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2333142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2333584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2334063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2334745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2335237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:12.2335900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2336658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:12.2337227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcushq78c
2022-12-01T11:16:12.2337841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcushq78c/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2338356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp31vqz1b
2022-12-01T11:16:12.2338894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp31vqz1b/_remote_module_non_scriptable.py
2022-12-01T11:16:12.2339412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2339897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:16:12.2340539Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpp31vqz1b'>
2022-12-01T11:16:12.2341002Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:16:12.2341588Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpcushq78c'>
2022-12-01T11:16:12.2342042Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:16:12.2342309Z ok (5.314s)
2022-12-01T11:16:12.2342722Z   test_zero_model_parallel_parameters_as_bucket_view_False (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2343457Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70075
2022-12-01T11:16:12.2343974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70076
2022-12-01T11:16:12.2344579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2345024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2345607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2346066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2346651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2347096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2347649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2348115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2348555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2349025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2349403Z skip: Need at least 4 CUDA devices (2.108s)
2022-12-01T11:16:12.2349865Z   test_zero_model_parallel_parameters_as_bucket_view_True (__main__.TestZeroRedundancyOptimizerDistributed)
2022-12-01T11:16:12.2350603Z Check that ZeRO works with model parallelism where the model's ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70143
2022-12-01T11:16:12.2351119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70144
2022-12-01T11:16:12.2351726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2352176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2352751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2353205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2353842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2354301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2354870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2355377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2355813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:12.2356286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2356662Z skip: Need at least 4 CUDA devices (2.008s)
2022-12-01T11:16:12.2357056Z   test_constructor (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2357626Z Check the robustness of the ZeroRedundancyOptimizer constructor by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70211
2022-12-01T11:16:12.2358335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2358771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2359350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2359818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2360240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2360731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2361393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2361791Z ok (2.007s)
2022-12-01T11:16:12.2362121Z   test_lr_scheduler (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2362670Z Check that a normal PyTorch ``lr_scheduler`` is usable with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70246
2022-12-01T11:16:12.2363349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2363805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2364361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2364828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2365267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2365735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2366393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2366796Z ok (3.509s)
2022-12-01T11:16:12.2367162Z   test_same_dense_param_type (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2367733Z Check that ZeroRedundancyOptimizer raises an exception if the input ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70288
2022-12-01T11:16:12.2368443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2368891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2369447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2369919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2370359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2370911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2371570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2372024Z ok (2.006s)
2022-12-01T11:16:12.2372369Z   test_state_dict (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2372924Z Check that ZeroRedundancyOptimizer exposes the expected state dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70323
2022-12-01T11:16:12.2373637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2374090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2375051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2375485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2375876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2376366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2377031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2377408Z ok (3.510s)
2022-12-01T11:16:12.2377778Z   test_step_with_extra_inner_key (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2378371Z Check that ZeroRedundancyOptimizer wrapping an optimizer that adds ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70365
2022-12-01T11:16:12.2379062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2379514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2380090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2380560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2380985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2381469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2382124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2382520Z ok (3.409s)
2022-12-01T11:16:12.2382858Z   test_step_with_kwargs (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2383419Z Check that the ``step(**kwargs)`` interface is properly exposed. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70407
2022-12-01T11:16:12.2384111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2384549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2385120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2385593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2386029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2386500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2387153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2387547Z ok (3.509s)
2022-12-01T11:16:12.2387893Z   test_step_without_closure (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2388545Z Check that the ``step()`` method (without closure) is handled as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70449
2022-12-01T11:16:12.2389245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2389773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2390338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2390803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2391240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2391726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2392364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2392757Z ok (3.509s)
2022-12-01T11:16:12.2393103Z   test_zero_grad (__main__.TestZeroRedundancyOptimizerSingleRank)
2022-12-01T11:16:12.2393619Z Check that the ``zero_grad`` method is properly handled. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70491
2022-12-01T11:16:12.2394431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:12.2394892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:12.2395472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:12.2395834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:12.2396215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:12.2396705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:12.2397347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-12-01T11:16:12.2397746Z ok (2.006s)
2022-12-01T11:16:12.2397898Z 
2022-12-01T11:16:12.2398177Z ----------------------------------------------------------------------
2022-12-01T11:16:12.2398510Z Ran 42 tests in 157.604s
2022-12-01T11:16:12.2398656Z 
2022-12-01T11:16:12.2398765Z OK (skipped=4)
2022-12-01T11:16:12.2398918Z 
2022-12-01T11:16:12.2399042Z Generating XML reports...
2022-12-01T11:16:12.2399758Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221201111334.xml
2022-12-01T11:16:12.2400736Z Generated XML report: test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221201111334.xml
2022-12-01T11:16:12.2401160Z 
2022-12-01T11:16:12.2401657Z ##[endgroup]
2022-12-01T11:16:12.2402324Z FINISHED PRINTING LOG FILE of distributed/optim/test_zero_redundancy_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_zero_redundancy_optimizer_ijiotkcl)
2022-12-01T11:16:12.2402718Z 
2022-12-01T11:16:12.2403042Z Running distributed/algorithms/quantization/test_quantization ... [2022-12-01 11:16:12.186898]
2022-12-01T11:16:12.2403368Z /usr/bin/mpiexec
2022-12-01T11:16:12.2403732Z MPI not available -- MPI backend tests will be skipped
2022-12-01T11:16:12.2404149Z Running distributed tests for the test backend with env init_method
2022-12-01T11:16:12.2404551Z Running distributed tests for the nccl backend with env init_method
2022-12-01T11:16:12.2404929Z Running distributed tests for the gloo backend with env init_method
2022-12-01T11:16:12.2405694Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:12.210021]
2022-12-01T11:16:12.2406266Z Running distributed tests for the ucc backend with env init_method
2022-12-01T11:16:12.2407077Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:12.211218]
2022-12-01T11:16:12.2408059Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:12.212443]
2022-12-01T11:16:12.2408952Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:12.213851]
2022-12-01T11:16:13.9562180Z 
2022-12-01T11:16:13.9562824Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:16:13.9563898Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_mkzqqldf)
2022-12-01T11:16:13.9564450Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:16:13.9564745Z 
2022-12-01T11:16:13.9564965Z ##[endgroup]
2022-12-01T11:16:13.9565777Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_mkzqqldf)
2022-12-01T11:16:13.9566220Z 
2022-12-01T11:16:13.9774918Z 
2022-12-01T11:16:13.9775386Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:16:13.9776399Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization__jcdymh5)
2022-12-01T11:16:13.9776956Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:16:13.9777140Z 
2022-12-01T11:16:13.9777361Z ##[endgroup]
2022-12-01T11:16:13.9778243Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization__jcdymh5)
2022-12-01T11:16:13.9778643Z 
2022-12-01T11:16:31.6675979Z 
2022-12-01T11:16:31.6676958Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:16:31.6679063Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_86al6m6l)
2022-12-01T11:16:31.6681498Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-12-01T11:16:31.6683361Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:31.6684051Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:31.6684803Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:31.6685518Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:31.6686267Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:31.6687025Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:31.6687715Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:16:31.6689015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6689926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6691128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6692534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6693064Z 
2022-12-01T11:16:31.6693350Z Running tests...
2022-12-01T11:16:31.6694877Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6696159Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:31.6697326Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:16:31.6698193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70770
2022-12-01T11:16:31.6698975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70771
2022-12-01T11:16:31.6700060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6700837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6701848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6703051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6704207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6705061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6706226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6707139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6707957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:31.6708781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:31.6709655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:31.6710531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:31.6711718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:31.6712975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:31.6713689Z ok (3.416s)
2022-12-01T11:16:31.6714021Z 
2022-12-01T11:16:31.6714488Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6715034Z Ran 1 test in 3.416s
2022-12-01T11:16:31.6715307Z 
2022-12-01T11:16:31.6715445Z OK
2022-12-01T11:16:31.6715674Z 
2022-12-01T11:16:31.6715882Z Generating XML reports...
2022-12-01T11:16:31.6717067Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111615.xml
2022-12-01T11:16:31.6718478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6719292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6720465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6721411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6721858Z 
2022-12-01T11:16:31.6722046Z Running tests...
2022-12-01T11:16:31.6722803Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6724003Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:31.6725052Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:16:31.6726107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70988
2022-12-01T11:16:31.6727015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70989
2022-12-01T11:16:31.6728346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6729376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6730556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6731491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6732646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6733496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6735249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6736205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6737048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:31.6738025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:31.6738990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:31.6739983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:31.6741321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:31.6742701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:31.6743493Z ok (3.542s)
2022-12-01T11:16:31.6743775Z 
2022-12-01T11:16:31.6744298Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6744916Z Ran 1 test in 3.543s
2022-12-01T11:16:31.6745230Z 
2022-12-01T11:16:31.6745403Z OK
2022-12-01T11:16:31.6745648Z 
2022-12-01T11:16:31.6745888Z Generating XML reports...
2022-12-01T11:16:31.6747191Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111620.xml
2022-12-01T11:16:31.6748725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6749612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6750768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6751693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6752159Z 
2022-12-01T11:16:31.6752366Z Running tests...
2022-12-01T11:16:31.6753140Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6754336Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:31.6755468Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.002s)
2022-12-01T11:16:31.6756011Z 
2022-12-01T11:16:31.6756514Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6757105Z Ran 1 test in 0.002s
2022-12-01T11:16:31.6757397Z 
2022-12-01T11:16:31.6757558Z OK (skipped=1)
2022-12-01T11:16:31.6757822Z 
2022-12-01T11:16:31.6758043Z Generating XML reports...
2022-12-01T11:16:31.6759360Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111625.xml
2022-12-01T11:16:31.6760901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6761925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6763129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6764180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6764657Z 
2022-12-01T11:16:31.6764832Z Running tests...
2022-12-01T11:16:31.6765622Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6766835Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:31.6767965Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-12-01T11:16:31.6768508Z 
2022-12-01T11:16:31.6769026Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6769650Z Ran 1 test in 0.001s
2022-12-01T11:16:31.6769977Z 
2022-12-01T11:16:31.6770179Z OK (skipped=1)
2022-12-01T11:16:31.6770485Z 
2022-12-01T11:16:31.6770699Z Generating XML reports...
2022-12-01T11:16:31.6772038Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111627.xml
2022-12-01T11:16:31.6773564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6774748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6775887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6776831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6777287Z 
2022-12-01T11:16:31.6777488Z Running tests...
2022-12-01T11:16:31.6778247Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6779459Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:31.6780619Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-12-01T11:16:31.6781228Z 
2022-12-01T11:16:31.6781760Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6782378Z Ran 1 test in 0.001s
2022-12-01T11:16:31.6782687Z 
2022-12-01T11:16:31.6782887Z OK (skipped=1)
2022-12-01T11:16:31.6783177Z 
2022-12-01T11:16:31.6783409Z Generating XML reports...
2022-12-01T11:16:31.6784727Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111629.xml
2022-12-01T11:16:31.6786239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:31.6787169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:31.6788325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:31.6789160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:31.6789608Z 
2022-12-01T11:16:31.6789809Z Running tests...
2022-12-01T11:16:31.6790582Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6791798Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:31.6792896Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-12-01T11:16:31.6793485Z 
2022-12-01T11:16:31.6794003Z ----------------------------------------------------------------------
2022-12-01T11:16:31.6794630Z Ran 1 test in 0.001s
2022-12-01T11:16:31.6794932Z 
2022-12-01T11:16:31.6795125Z OK (skipped=1)
2022-12-01T11:16:31.6795381Z 
2022-12-01T11:16:31.6795619Z Generating XML reports...
2022-12-01T11:16:31.6797125Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111631.xml
2022-12-01T11:16:31.6797955Z 
2022-12-01T11:16:31.6798689Z ##[endgroup]
2022-12-01T11:16:31.6800102Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_86al6m6l)
2022-12-01T11:16:31.6800970Z 
2022-12-01T11:16:49.1983737Z 
2022-12-01T11:16:49.1984489Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:16:49.1985514Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_i830th65)
2022-12-01T11:16:49.1986648Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-12-01T11:16:49.1987521Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:49.1987900Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:49.1988292Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:49.1988637Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:49.1989007Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:49.1989387Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:16:49.1989710Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:16:49.1990415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.1990812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.1991401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.1991935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.1992160Z 
2022-12-01T11:16:49.1992275Z Running tests...
2022-12-01T11:16:49.1992638Z ----------------------------------------------------------------------
2022-12-01T11:16:49.1993232Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:49.1993796Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-12-01T11:16:49.1994092Z 
2022-12-01T11:16:49.1994356Z ----------------------------------------------------------------------
2022-12-01T11:16:49.1994690Z Ran 1 test in 0.001s
2022-12-01T11:16:49.1994832Z 
2022-12-01T11:16:49.1994945Z OK (skipped=1)
2022-12-01T11:16:49.1995101Z 
2022-12-01T11:16:49.1995227Z Generating XML reports...
2022-12-01T11:16:49.1995894Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111615.xml
2022-12-01T11:16:49.1996641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.1997195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.1997758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.1998238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.1998470Z 
2022-12-01T11:16:49.1998582Z Running tests...
2022-12-01T11:16:49.1998988Z ----------------------------------------------------------------------
2022-12-01T11:16:49.1999840Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:49.2000469Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-12-01T11:16:49.2000889Z 
2022-12-01T11:16:49.2001184Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2001514Z Ran 1 test in 0.001s
2022-12-01T11:16:49.2001685Z 
2022-12-01T11:16:49.2001797Z OK (skipped=1)
2022-12-01T11:16:49.2001959Z 
2022-12-01T11:16:49.2002087Z Generating XML reports...
2022-12-01T11:16:49.2002773Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111616.xml
2022-12-01T11:16:49.2003549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2004027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2004641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2005148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2005379Z 
2022-12-01T11:16:49.2005496Z Running tests...
2022-12-01T11:16:49.2005921Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2006568Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:49.2007120Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:16:49.2007623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70911
2022-12-01T11:16:49.2008108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70912
2022-12-01T11:16:49.2008752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2009215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2009822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2010326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2010928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2011407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2012019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2012518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2012967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:49.2013476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:49.2013994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:49.2015136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:49.2015843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2016385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:16:49.2017033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2017548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:16:49.2018209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2019001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2019413Z ok (6.256s)
2022-12-01T11:16:49.2019630Z 
2022-12-01T11:16:49.2019887Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2020218Z Ran 1 test in 6.256s
2022-12-01T11:16:49.2020381Z 
2022-12-01T11:16:49.2020477Z OK
2022-12-01T11:16:49.2020612Z 
2022-12-01T11:16:49.2020717Z Generating XML reports...
2022-12-01T11:16:49.2021383Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111618.xml
2022-12-01T11:16:49.2022147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2022602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2023171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2023641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2023873Z 
2022-12-01T11:16:49.2023989Z Running tests...
2022-12-01T11:16:49.2024396Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2024980Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:49.2025527Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:16:49.2026002Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71174
2022-12-01T11:16:49.2026433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71175
2022-12-01T11:16:49.2027038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2027488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2028065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2028524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2029098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2029543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2030148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2030617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2031058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:49.2031534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:49.2032006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:49.2032501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:49.2033160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2033693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:16:49.2034323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2034846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:16:49.2035493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2036228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2036637Z ok (6.091s)
2022-12-01T11:16:49.2036787Z 
2022-12-01T11:16:49.2037057Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2037440Z Ran 1 test in 6.091s
2022-12-01T11:16:49.2037584Z 
2022-12-01T11:16:49.2037678Z OK
2022-12-01T11:16:49.2037813Z 
2022-12-01T11:16:49.2037937Z Generating XML reports...
2022-12-01T11:16:49.2038599Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111626.xml
2022-12-01T11:16:49.2039340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2039790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2040367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2040839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2041069Z 
2022-12-01T11:16:49.2041160Z Running tests...
2022-12-01T11:16:49.2041568Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2042176Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:49.2042736Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:16:49.2043202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71358
2022-12-01T11:16:49.2043653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71359
2022-12-01T11:16:49.2044266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2044696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2045274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2045743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2046320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2046748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2047321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2047784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2048204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:49.2048698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:49.2049185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:49.2049675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:49.2050316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2050846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:16:49.2051492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2052028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:16:49.2052660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2053343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2053799Z ok (6.075s)
2022-12-01T11:16:49.2053961Z 
2022-12-01T11:16:49.2054234Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2055040Z Ran 1 test in 6.075s
2022-12-01T11:16:49.2055207Z 
2022-12-01T11:16:49.2055301Z OK
2022-12-01T11:16:49.2055436Z 
2022-12-01T11:16:49.2055561Z Generating XML reports...
2022-12-01T11:16:49.2056210Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111634.xml
2022-12-01T11:16:49.2056973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2057424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2057999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2058455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2058690Z 
2022-12-01T11:16:49.2058799Z Running tests...
2022-12-01T11:16:49.2059203Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2059794Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:16:49.2060348Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:16:49.2060832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71477
2022-12-01T11:16:49.2061282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71478
2022-12-01T11:16:49.2061867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2062317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2062898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2063353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2063931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:16:49.2064379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:16:49.2064954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:16:49.2065403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:16:49.2065843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:16:49.2066335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:16:49.2066801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:16:49.2067283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:16:49.2067942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2068635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:16:49.2069146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:16:49.2069636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:16:49.2070283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2070973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:16:49.2071350Z ok (6.076s)
2022-12-01T11:16:49.2071589Z 
2022-12-01T11:16:49.2071884Z ----------------------------------------------------------------------
2022-12-01T11:16:49.2072223Z Ran 1 test in 6.076s
2022-12-01T11:16:49.2072438Z 
2022-12-01T11:16:49.2072514Z OK
2022-12-01T11:16:49.2072650Z 
2022-12-01T11:16:49.2072777Z Generating XML reports...
2022-12-01T11:16:49.2073438Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111642.xml
2022-12-01T11:16:49.2073842Z 
2022-12-01T11:16:49.2074184Z ##[endgroup]
2022-12-01T11:16:49.2074899Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_i830th65)
2022-12-01T11:16:49.2075337Z 
2022-12-01T11:16:49.2215035Z Running distributed tests for the test backend with file init_method
2022-12-01T11:16:49.2236141Z Running distributed tests for the nccl backend with file init_method
2022-12-01T11:16:49.2248765Z Running distributed tests for the gloo backend with file init_method
2022-12-01T11:16:49.2252067Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:49.224788]
2022-12-01T11:16:49.2261295Z Running distributed tests for the ucc backend with file init_method
2022-12-01T11:16:49.2263738Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:49.226011]
2022-12-01T11:16:49.2276916Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:49.227253]
2022-12-01T11:16:49.2289922Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:16:49.228539]
2022-12-01T11:16:50.9574160Z 
2022-12-01T11:16:50.9575219Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:16:50.9576558Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ax0l_yhm)
2022-12-01T11:16:50.9577271Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:16:50.9577637Z 
2022-12-01T11:16:50.9578133Z ##[endgroup]
2022-12-01T11:16:50.9579191Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ax0l_yhm)
2022-12-01T11:16:50.9579631Z 
2022-12-01T11:16:50.9587837Z 
2022-12-01T11:16:50.9588339Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:16:50.9589399Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_v0knp43l)
2022-12-01T11:16:50.9590101Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:16:50.9590470Z 
2022-12-01T11:16:50.9591020Z ##[endgroup]
2022-12-01T11:16:50.9592473Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_v0knp43l)
2022-12-01T11:16:50.9593003Z 
2022-12-01T11:17:08.7825655Z 
2022-12-01T11:17:08.7826488Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:17:08.7828502Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_zhv5zk8c)
2022-12-01T11:17:08.7831040Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-12-01T11:17:08.7832082Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:08.7832518Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:08.7832959Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:08.7833314Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:08.7833690Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:08.7834072Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:08.7834405Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:17:08.7835092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7835642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7836368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7837066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7837344Z 
2022-12-01T11:17:08.7837437Z Running tests...
2022-12-01T11:17:08.7837919Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7838615Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:08.7839250Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:17:08.7839800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71802
2022-12-01T11:17:08.7840334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71803
2022-12-01T11:17:08.7841029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7841501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7856257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7856750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7857350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7857810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7858400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7858869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7859319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:17:08.7859809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:17:08.7860294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:17:08.7860805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:17:08.7861477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:08.7862179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:08.7862562Z ok (3.392s)
2022-12-01T11:17:08.7862713Z 
2022-12-01T11:17:08.7863156Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7863512Z Ran 1 test in 3.392s
2022-12-01T11:17:08.7863675Z 
2022-12-01T11:17:08.7863768Z OK
2022-12-01T11:17:08.7863959Z 
2022-12-01T11:17:08.7864086Z Generating XML reports...
2022-12-01T11:17:08.7864765Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111652.xml
2022-12-01T11:17:08.7865538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7865983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7866566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7867041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7867275Z 
2022-12-01T11:17:08.7867381Z Running tests...
2022-12-01T11:17:08.7867775Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7868381Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:08.7868949Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:17:08.7869413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72020
2022-12-01T11:17:08.7869873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72021
2022-12-01T11:17:08.7870485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7870945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7871506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7871980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7872566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7873018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7873576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7874047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7874492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:17:08.7874959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:17:08.7875449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:17:08.7875958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:17:08.7876630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:08.7877312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:08.7877718Z ok (3.532s)
2022-12-01T11:17:08.7877867Z 
2022-12-01T11:17:08.7878134Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7878444Z Ran 1 test in 3.532s
2022-12-01T11:17:08.7878606Z 
2022-12-01T11:17:08.7878697Z OK
2022-12-01T11:17:08.7878830Z 
2022-12-01T11:17:08.7878954Z Generating XML reports...
2022-12-01T11:17:08.7879621Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111657.xml
2022-12-01T11:17:08.7880368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7880893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7881486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7882009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7882244Z 
2022-12-01T11:17:08.7882352Z Running tests...
2022-12-01T11:17:08.7882757Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7883367Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:08.7883919Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-12-01T11:17:08.7884211Z 
2022-12-01T11:17:08.7884473Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7884798Z Ran 1 test in 0.001s
2022-12-01T11:17:08.7884958Z 
2022-12-01T11:17:08.7885065Z OK (skipped=1)
2022-12-01T11:17:08.7885201Z 
2022-12-01T11:17:08.7885329Z Generating XML reports...
2022-12-01T11:17:08.7885994Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111702.xml
2022-12-01T11:17:08.7886763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7887199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7887780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7888257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7888491Z 
2022-12-01T11:17:08.7888600Z Running tests...
2022-12-01T11:17:08.7888989Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7889595Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:08.7890168Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-12-01T11:17:08.7890461Z 
2022-12-01T11:17:08.7890728Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7891034Z Ran 1 test in 0.001s
2022-12-01T11:17:08.7891196Z 
2022-12-01T11:17:08.7891301Z OK (skipped=1)
2022-12-01T11:17:08.7891455Z 
2022-12-01T11:17:08.7891576Z Generating XML reports...
2022-12-01T11:17:08.7892224Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111704.xml
2022-12-01T11:17:08.7892988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7893447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7894033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7894996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7895235Z 
2022-12-01T11:17:08.7895351Z Running tests...
2022-12-01T11:17:08.7895762Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7896353Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:08.7896943Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-12-01T11:17:08.7897251Z 
2022-12-01T11:17:08.7897515Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7897841Z Ran 1 test in 0.001s
2022-12-01T11:17:08.7897984Z 
2022-12-01T11:17:08.7898092Z OK (skipped=1)
2022-12-01T11:17:08.7898244Z 
2022-12-01T11:17:08.7898367Z Generating XML reports...
2022-12-01T11:17:08.7899134Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111706.xml
2022-12-01T11:17:08.7899904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:08.7900440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:08.7901024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:08.7901498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:08.7901731Z 
2022-12-01T11:17:08.7901822Z Running tests...
2022-12-01T11:17:08.7902226Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7902833Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:08.7903428Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-12-01T11:17:08.7903718Z 
2022-12-01T11:17:08.7903981Z ----------------------------------------------------------------------
2022-12-01T11:17:08.7904309Z Ran 1 test in 0.001s
2022-12-01T11:17:08.7904469Z 
2022-12-01T11:17:08.7904575Z OK (skipped=1)
2022-12-01T11:17:08.7904730Z 
2022-12-01T11:17:08.7904835Z Generating XML reports...
2022-12-01T11:17:08.7905499Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111708.xml
2022-12-01T11:17:08.7905906Z 
2022-12-01T11:17:08.7906274Z ##[endgroup]
2022-12-01T11:17:08.7906981Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_zhv5zk8c)
2022-12-01T11:17:08.7907431Z 
2022-12-01T11:17:26.4418659Z 
2022-12-01T11:17:26.4419316Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-12-01T11:17:26.4420349Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_x5xj8gwd)
2022-12-01T11:17:26.4421592Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-12-01T11:17:26.4422380Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:26.4422724Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:26.4423099Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:26.4423475Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:26.4423961Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:26.4424277Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-12-01T11:17:26.4424635Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:17:26.4425352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4425828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4426409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4426875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4427116Z 
2022-12-01T11:17:26.4427217Z Running tests...
2022-12-01T11:17:26.4427585Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4428458Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:26.4429045Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-12-01T11:17:26.4429493Z 
2022-12-01T11:17:26.4429750Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4430329Z Ran 1 test in 0.001s
2022-12-01T11:17:26.4430619Z 
2022-12-01T11:17:26.4430872Z OK (skipped=1)
2022-12-01T11:17:26.4431206Z 
2022-12-01T11:17:26.4431372Z Generating XML reports...
2022-12-01T11:17:26.4432690Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111652.xml
2022-12-01T11:17:26.4434279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4434727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4435976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4436601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4436830Z 
2022-12-01T11:17:26.4436946Z Running tests...
2022-12-01T11:17:26.4437341Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4438206Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:26.4438782Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2022-12-01T11:17:26.4439038Z 
2022-12-01T11:17:26.4439569Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4440284Z Ran 1 test in 0.001s
2022-12-01T11:17:26.4440563Z 
2022-12-01T11:17:26.4440763Z OK (skipped=1)
2022-12-01T11:17:26.4440940Z 
2022-12-01T11:17:26.4441075Z Generating XML reports...
2022-12-01T11:17:26.4441729Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111653.xml
2022-12-01T11:17:26.4442509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4442950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4443533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4444014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4444251Z 
2022-12-01T11:17:26.4444360Z Running tests...
2022-12-01T11:17:26.4444748Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4445359Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:26.4445921Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:17:26.4446387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71944
2022-12-01T11:17:26.4446843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71945
2022-12-01T11:17:26.4447464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4447926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4448495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4448966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4449559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4450007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4450678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4451170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4451683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:17:26.4452170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:17:26.4452669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:17:26.4453194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:17:26.4453839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4454867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:17:26.4455561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4456100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:17:26.4456765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4457446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4457882Z ok (6.424s)
2022-12-01T11:17:26.4458004Z 
2022-12-01T11:17:26.4458269Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4458582Z Ran 1 test in 6.424s
2022-12-01T11:17:26.4458746Z 
2022-12-01T11:17:26.4458838Z OK
2022-12-01T11:17:26.4459003Z 
2022-12-01T11:17:26.4459128Z Generating XML reports...
2022-12-01T11:17:26.4459768Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111655.xml
2022-12-01T11:17:26.4460519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4460982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4461567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4462028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4462263Z 
2022-12-01T11:17:26.4462372Z Running tests...
2022-12-01T11:17:26.4462775Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4463381Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:26.4463918Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:17:26.4464401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72206
2022-12-01T11:17:26.4464856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72207
2022-12-01T11:17:26.4465460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4465916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4466498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4466977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4467542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4467992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4468687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4469182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4469610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:17:26.4470186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:17:26.4470688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:17:26.4471160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:17:26.4471829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4472376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:17:26.4473031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4473561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:17:26.4474214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4474904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4475306Z ok (6.130s)
2022-12-01T11:17:26.4475436Z 
2022-12-01T11:17:26.4475728Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4476085Z Ran 1 test in 6.130s
2022-12-01T11:17:26.4476245Z 
2022-12-01T11:17:26.4476340Z OK
2022-12-01T11:17:26.4476474Z 
2022-12-01T11:17:26.4476600Z Generating XML reports...
2022-12-01T11:17:26.4477253Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111704.xml
2022-12-01T11:17:26.4478031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4478490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4479075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4479630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4479767Z 
2022-12-01T11:17:26.4479878Z Running tests...
2022-12-01T11:17:26.4480282Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4480976Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:26.4481438Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:17:26.4482036Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72390
2022-12-01T11:17:26.4482389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72391
2022-12-01T11:17:26.4482990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4483454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4484036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4484629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4485085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4485538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4486114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4486632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4487143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:17:26.4487651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:17:26.4488245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:17:26.4488623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:17:26.4489309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4490068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:17:26.4490698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4491233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:17:26.4491901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4492685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4493149Z ok (5.981s)
2022-12-01T11:17:26.4493291Z 
2022-12-01T11:17:26.4493565Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4493864Z Ran 1 test in 5.982s
2022-12-01T11:17:26.4494036Z 
2022-12-01T11:17:26.4494129Z OK
2022-12-01T11:17:26.4494252Z 
2022-12-01T11:17:26.4495032Z Generating XML reports...
2022-12-01T11:17:26.4495729Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111712.xml
2022-12-01T11:17:26.4496502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4496945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4497534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4498019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4498253Z 
2022-12-01T11:17:26.4498361Z Running tests...
2022-12-01T11:17:26.4498748Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4499357Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2022-12-01T11:17:26.4499922Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:17:26.4500393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72509
2022-12-01T11:17:26.4500855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72510
2022-12-01T11:17:26.4501467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4501930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4502497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4502970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4503559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:17:26.4504006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:17:26.4504568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:17:26.4505142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:17:26.4505603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:17:26.4506083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:17:26.4506664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:17:26.4507157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:17:26.4507834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4508362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:17:26.4509022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:17:26.4509562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:17:26.4510220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4510900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:17:26.4511299Z ok (6.069s)
2022-12-01T11:17:26.4511447Z 
2022-12-01T11:17:26.4511715Z ----------------------------------------------------------------------
2022-12-01T11:17:26.4512025Z Ran 1 test in 6.069s
2022-12-01T11:17:26.4512185Z 
2022-12-01T11:17:26.4512282Z OK
2022-12-01T11:17:26.4512416Z 
2022-12-01T11:17:26.4512540Z Generating XML reports...
2022-12-01T11:17:26.4513210Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111719.xml
2022-12-01T11:17:26.4513604Z 
2022-12-01T11:17:26.4513971Z ##[endgroup]
2022-12-01T11:17:26.4514698Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_x5xj8gwd)
2022-12-01T11:17:26.4515154Z 
2022-12-01T11:17:26.4515433Z Running distributed/test_c10d_spawn_nccl ... [2022-12-01 11:17:26.448824]
2022-12-01T11:17:26.4516128Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:17:26.449154]
2022-12-01T11:18:43.2702144Z 
2022-12-01T11:18:43.2702654Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2022-12-01T11:18:43.2705669Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_9_59q8vm)
2022-12-01T11:18:43.2706640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ipdywcg
2022-12-01T11:18:43.2707258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ipdywcg/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2707685Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2708020Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:18:43.2708378Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:18:43.2709970Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2022-12-01T11:18:43.2711642Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2712019Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2712676Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2712991Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2713422Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2713831Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2714238Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2714634Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2715107Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2022-12-01T11:18:43.2715867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2716321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2716850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2717431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2717914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdbef59x9
2022-12-01T11:18:43.2718451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdbef59x9/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2718806Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2719061Z 
2022-12-01T11:18:43.2719170Z Running tests...
2022-12-01T11:18:43.2719564Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2720206Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2720765Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72662
2022-12-01T11:18:43.2721321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72663
2022-12-01T11:18:43.2721931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2722402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2722997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2723606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2724210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2724600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2725276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2726065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2726556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsmnnn36
2022-12-01T11:18:43.2727106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsmnnn36/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2727549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphkdsad6h
2022-12-01T11:18:43.2728064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphkdsad6h/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2728490Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2728886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2729417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2729823Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2730296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2730793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2731606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2732312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2732695Z ok (4.826s)
2022-12-01T11:18:43.2732848Z 
2022-12-01T11:18:43.2733153Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2733450Z Ran 1 test in 4.827s
2022-12-01T11:18:43.2733613Z 
2022-12-01T11:18:43.2733707Z OK
2022-12-01T11:18:43.2733822Z 
2022-12-01T11:18:43.2733951Z Generating XML reports...
2022-12-01T11:18:43.2735190Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111732.xml
2022-12-01T11:18:43.2735936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2736386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2736974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2737456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2737933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmx47ofkr
2022-12-01T11:18:43.2738467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmx47ofkr/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2738908Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2739109Z 
2022-12-01T11:18:43.2739247Z Running tests...
2022-12-01T11:18:43.2739612Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2740164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2740759Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72781
2022-12-01T11:18:43.2741323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72782
2022-12-01T11:18:43.2741924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2742381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2743006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2743447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2744066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2744525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2745107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2745584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2746043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eb7dcxa
2022-12-01T11:18:43.2746638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eb7dcxa/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2747139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp858rzwoo
2022-12-01T11:18:43.2747666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp858rzwoo/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2748105Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2748520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2749012Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2749466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2750038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2750544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2751191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2751891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2752840Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:18:43.2753401Z   warnings.warn(
2022-12-01T11:18:43.2754193Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:18:43.2754711Z   warnings.warn(
2022-12-01T11:18:43.2755498Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T11:18:43.2756056Z   warnings.warn(
2022-12-01T11:18:43.2756802Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2849: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-12-01T11:18:43.2757355Z   warnings.warn(
2022-12-01T11:18:43.2757594Z ok (4.727s)
2022-12-01T11:18:43.2757749Z 
2022-12-01T11:18:43.2758020Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2758331Z Ran 1 test in 4.728s
2022-12-01T11:18:43.2758500Z 
2022-12-01T11:18:43.2758595Z OK
2022-12-01T11:18:43.2758730Z 
2022-12-01T11:18:43.2758854Z Generating XML reports...
2022-12-01T11:18:43.2759489Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111740.xml
2022-12-01T11:18:43.2760287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2760707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2761294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2761756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2762235Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewcukz5s
2022-12-01T11:18:43.2762811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewcukz5s/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2763215Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2763420Z 
2022-12-01T11:18:43.2763531Z Running tests...
2022-12-01T11:18:43.2763938Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2764487Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2765049Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72900
2022-12-01T11:18:43.2765600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72901
2022-12-01T11:18:43.2766218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2766784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2767330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2767872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2768464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2768900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2769483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2769958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2770430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mckav47
2022-12-01T11:18:43.2770968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mckav47/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2771507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp687zhgd6
2022-12-01T11:18:43.2772054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp687zhgd6/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2772466Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2772883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2773384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2773793Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2774186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2774902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2775584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2776268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2776674Z ok (4.827s)
2022-12-01T11:18:43.2776825Z 
2022-12-01T11:18:43.2777130Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2777425Z Ran 1 test in 4.828s
2022-12-01T11:18:43.2777570Z 
2022-12-01T11:18:43.2777665Z OK
2022-12-01T11:18:43.2777838Z 
2022-12-01T11:18:43.2777956Z Generating XML reports...
2022-12-01T11:18:43.2778578Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111748.xml
2022-12-01T11:18:43.2779315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2779780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2780371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2780852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2781314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fn0x36s
2022-12-01T11:18:43.2781857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fn0x36s/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2782292Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2782490Z 
2022-12-01T11:18:43.2782600Z Running tests...
2022-12-01T11:18:43.2782988Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2783539Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2784132Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73021
2022-12-01T11:18:43.2784769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73022
2022-12-01T11:18:43.2785406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2785943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2786531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2786996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2787636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2788043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2788604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2789084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2789558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1wzxess
2022-12-01T11:18:43.2790116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1wzxess/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2790646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzd5r7gqj
2022-12-01T11:18:43.2791188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzd5r7gqj/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2791622Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2791930Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2792342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2792842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2793337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2793814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2794491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2795195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2795600Z ok (4.828s)
2022-12-01T11:18:43.2795731Z 
2022-12-01T11:18:43.2796002Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2796334Z Ran 1 test in 4.829s
2022-12-01T11:18:43.2796497Z 
2022-12-01T11:18:43.2796591Z OK
2022-12-01T11:18:43.2796706Z 
2022-12-01T11:18:43.2796831Z Generating XML reports...
2022-12-01T11:18:43.2797482Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111757.xml
2022-12-01T11:18:43.2798247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2798709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2799279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2799759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2800237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpnpdvgm
2022-12-01T11:18:43.2800773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpnpdvgm/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2801209Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2801408Z 
2022-12-01T11:18:43.2801518Z Running tests...
2022-12-01T11:18:43.2801926Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2802524Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2803122Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73142
2022-12-01T11:18:43.2803734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73143
2022-12-01T11:18:43.2804337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2804797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2805378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2805856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2806423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2806885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2807462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2807923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2808396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuqyadqpf
2022-12-01T11:18:43.2808953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuqyadqpf/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2809498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqbhm286
2022-12-01T11:18:43.2810022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqbhm286/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2810559Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2810866Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2811275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2811778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2812255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2812754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2813425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2814128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2814809Z ok (4.826s)
2022-12-01T11:18:43.2814962Z 
2022-12-01T11:18:43.2815195Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2815530Z Ran 1 test in 4.827s
2022-12-01T11:18:43.2815693Z 
2022-12-01T11:18:43.2815768Z OK
2022-12-01T11:18:43.2815903Z 
2022-12-01T11:18:43.2816034Z Generating XML reports...
2022-12-01T11:18:43.2816687Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111805.xml
2022-12-01T11:18:43.2817444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2817894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2818468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2818962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2819420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmoxuzu8k
2022-12-01T11:18:43.2819954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmoxuzu8k/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2820394Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2820593Z 
2022-12-01T11:18:43.2820797Z Running tests...
2022-12-01T11:18:43.2821203Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2821845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2822436Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73261
2022-12-01T11:18:43.2822990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73262
2022-12-01T11:18:43.2823590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2824046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2824631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2825098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2825689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2826146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2826724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2827225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2827656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprl2fa2qf
2022-12-01T11:18:43.2828273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprl2fa2qf/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2828749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9zl3084
2022-12-01T11:18:43.2829276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9zl3084/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2829713Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2830127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2830613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2831025Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2831436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2831917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2832593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2833298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2833701Z ok (4.727s)
2022-12-01T11:18:43.2833851Z 
2022-12-01T11:18:43.2834108Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2834440Z Ran 1 test in 4.728s
2022-12-01T11:18:43.2834604Z 
2022-12-01T11:18:43.2834702Z OK
2022-12-01T11:18:43.2834836Z 
2022-12-01T11:18:43.2834961Z Generating XML reports...
2022-12-01T11:18:43.2835598Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111813.xml
2022-12-01T11:18:43.2836352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2836814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2837382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2837865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2838407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpboyr3t3w
2022-12-01T11:18:43.2838968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpboyr3t3w/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2839444Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2839644Z 
2022-12-01T11:18:43.2839758Z Running tests...
2022-12-01T11:18:43.2840168Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2840697Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2841280Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73380
2022-12-01T11:18:43.2841836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73381
2022-12-01T11:18:43.2842455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2842901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2843486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2844023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2844592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2845045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2845629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2846102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2846617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9b4r3_0
2022-12-01T11:18:43.2847498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9b4r3_0/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2848192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxh5adqy
2022-12-01T11:18:43.2848745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxh5adqy/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2849168Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2849588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2850087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2850478Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2850887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2851387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2852073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2852764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2853171Z ok (4.726s)
2022-12-01T11:18:43.2853325Z 
2022-12-01T11:18:43.2853597Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2853910Z Ran 1 test in 4.727s
2022-12-01T11:18:43.2854073Z 
2022-12-01T11:18:43.2854167Z OK
2022-12-01T11:18:43.2854300Z 
2022-12-01T11:18:43.2854789Z Generating XML reports...
2022-12-01T11:18:43.2855415Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111821.xml
2022-12-01T11:18:43.2856154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2856619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2857318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2857804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2858281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje6rs3wu
2022-12-01T11:18:43.2858924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje6rs3wu/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2859362Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2859542Z 
2022-12-01T11:18:43.2859651Z Running tests...
2022-12-01T11:18:43.2860063Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2860608Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2861186Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73499
2022-12-01T11:18:43.2861756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73500
2022-12-01T11:18:43.2862382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2862848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2863417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2863895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2864487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2864941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2865502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2865974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2866456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73lktq6o
2022-12-01T11:18:43.2866989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73lktq6o/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2867539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp46a7z1
2022-12-01T11:18:43.2868089Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp46a7z1/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2868523Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2868916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2869417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2869829Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2870219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2870716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2871390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2872098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2872482Z ok (4.827s)
2022-12-01T11:18:43.2872632Z 
2022-12-01T11:18:43.2872906Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2873241Z Ran 1 test in 4.828s
2022-12-01T11:18:43.2873405Z 
2022-12-01T11:18:43.2873480Z OK
2022-12-01T11:18:43.2873615Z 
2022-12-01T11:18:43.2873741Z Generating XML reports...
2022-12-01T11:18:43.2874393Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111829.xml
2022-12-01T11:18:43.2875210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2875669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2876259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2876802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2877260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkuzqsiou
2022-12-01T11:18:43.2877814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkuzqsiou/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2878253Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2878452Z 
2022-12-01T11:18:43.2878561Z Running tests...
2022-12-01T11:18:43.2878953Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2879497Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-12-01T11:18:43.2880115Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73618
2022-12-01T11:18:43.2880703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73619
2022-12-01T11:18:43.2881302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2881759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2882343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2882805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2883392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:18:43.2883845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:18:43.2884432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:18:43.2884889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:18:43.2885376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcj8mravi
2022-12-01T11:18:43.2885933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcj8mravi/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2886457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponst8nfc
2022-12-01T11:18:43.2887004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponst8nfc/_remote_module_non_scriptable.py
2022-12-01T11:18:43.2887443Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2887860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:18:43.2888343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:18:43.2888754Z INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:18:43.2889163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:18:43.2889643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:18:43.2890321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2891025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:18:43.2891428Z ok (4.727s)
2022-12-01T11:18:43.2891578Z 
2022-12-01T11:18:43.2891828Z ----------------------------------------------------------------------
2022-12-01T11:18:43.2892160Z Ran 1 test in 4.727s
2022-12-01T11:18:43.2892322Z 
2022-12-01T11:18:43.2892415Z OK
2022-12-01T11:18:43.2892547Z 
2022-12-01T11:18:43.2892653Z Generating XML reports...
2022-12-01T11:18:43.2893368Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111837.xml
2022-12-01T11:18:43.2893820Z 
2022-12-01T11:18:43.2894169Z ##[endgroup]
2022-12-01T11:18:43.2895196Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_9_59q8vm)
2022-12-01T11:18:43.2895504Z 
2022-12-01T11:18:43.2895774Z Running distributed/algorithms/test_join ... [2022-12-01 11:18:43.270455]
2022-12-01T11:18:43.2896460Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:18:43.270725]
2022-12-01T11:19:26.8659857Z 
2022-12-01T11:19:26.8660696Z Expand the folded group to see the log file of distributed/algorithms/test_join
2022-12-01T11:19:26.8661817Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_bx9i47a_)
2022-12-01T11:19:26.8662201Z 
2022-12-01T11:19:26.8662297Z Running tests...
2022-12-01T11:19:26.8666717Z ----------------------------------------------------------------------
2022-12-01T11:19:26.8667665Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2022-12-01T11:19:26.8668350Z   test_join_kwargs (__main__.TestJoin)
2022-12-01T11:19:26.8669063Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:19:26.8669829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73737
2022-12-01T11:19:26.8670535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73738
2022-12-01T11:19:26.8671537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8672293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8673187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8678296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8678998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8679496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8680091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8680514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8680981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8681504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8682000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8682492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8683205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8683904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8684280Z ok (6.037s)
2022-12-01T11:19:26.8684551Z   test_multiple_joinable_disable (__main__.TestJoin)
2022-12-01T11:19:26.8685076Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73817
2022-12-01T11:19:26.8685628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73818
2022-12-01T11:19:26.8686250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8686936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8687600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8688229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8688843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8689404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8689971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8690477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8690919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8691492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8691956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8692479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8693183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8693959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8694799Z ok (4.512s)
2022-12-01T11:19:26.8695135Z   test_multiple_joinables (__main__.TestJoin)
2022-12-01T11:19:26.8695739Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73897
2022-12-01T11:19:26.8696313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73898
2022-12-01T11:19:26.8696911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8697356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8697962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8698367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8698949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8699429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8700007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8700519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8700933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8701432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8701915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8702410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8703101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8703773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8704165Z ok (4.512s)
2022-12-01T11:19:26.8704448Z   test_multiple_joinables_throw (__main__.TestJoin)
2022-12-01T11:19:26.8705037Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73977
2022-12-01T11:19:26.8705577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73978
2022-12-01T11:19:26.8706188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8706762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8707313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8707772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8708354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8708806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8709365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8709843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8710325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8710794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8711272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8711773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8712429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8713127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8713507Z ok (4.412s)
2022-12-01T11:19:26.8713778Z   test_single_joinable (__main__.TestJoin)
2022-12-01T11:19:26.8714398Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74057
2022-12-01T11:19:26.8714930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74058
2022-12-01T11:19:26.8715545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8716000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8716584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8717044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8717626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8718076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8718640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8719124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8719563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8720062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8720541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8721029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8721696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8722397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8722863Z ok (4.412s)
2022-12-01T11:19:26.8723132Z   test_single_joinable_disable (__main__.TestJoin)
2022-12-01T11:19:26.8723613Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74137
2022-12-01T11:19:26.8724174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74138
2022-12-01T11:19:26.8724796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8725250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8725828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8726285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8726899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8727326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8727883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8728361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8728804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8729305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8729783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8730275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8730942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8731642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8732025Z ok (4.512s)
2022-12-01T11:19:26.8732317Z   test_single_joinable_main_hooks (__main__.TestJoin)
2022-12-01T11:19:26.8732794Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74217
2022-12-01T11:19:26.8733298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74218
2022-12-01T11:19:26.8733916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8734550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8735140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8735601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8736189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8736643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8737208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8737680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8738124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8738622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8739103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8739595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8740348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8741066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8741521Z ok (4.438s)
2022-12-01T11:19:26.8741810Z   test_single_joinable_post_hooks (__main__.TestJoin)
2022-12-01T11:19:26.8742416Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74297
2022-12-01T11:19:26.8742926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74298
2022-12-01T11:19:26.8743537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8743994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8744578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8745040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8745626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8746082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8746642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8747114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8747558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8748054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8748531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8749069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8749740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8750440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8750824Z ok (4.413s)
2022-12-01T11:19:26.8751105Z   test_single_joinable_throw (__main__.TestJoin)
2022-12-01T11:19:26.8751584Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74377
2022-12-01T11:19:26.8752085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74378
2022-12-01T11:19:26.8752701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8753160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8753744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8754214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8754798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:26.8755251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:26.8755812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:26.8756288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:26.8756731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:19:26.8757228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:19:26.8757768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:19:26.8758272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:19:26.8758998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8759698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:19:26.8760083Z ok (4.512s)
2022-12-01T11:19:26.8760235Z 
2022-12-01T11:19:26.8760505Z ----------------------------------------------------------------------
2022-12-01T11:19:26.8760836Z Ran 9 tests in 41.760s
2022-12-01T11:19:26.8761002Z 
2022-12-01T11:19:26.8761077Z OK
2022-12-01T11:19:26.8761210Z 
2022-12-01T11:19:26.8761334Z Generating XML reports...
2022-12-01T11:19:26.8761904Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221201111844.xml
2022-12-01T11:19:26.8762249Z 
2022-12-01T11:19:26.8762714Z ##[endgroup]
2022-12-01T11:19:26.8763312Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_bx9i47a_)
2022-12-01T11:19:26.8763681Z 
2022-12-01T11:19:26.8763978Z Running distributed/elastic/multiprocessing/api_test ... [2022-12-01 11:19:26.866134]
2022-12-01T11:19:26.8764712Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:19:26.866411]
2022-12-01T11:19:51.7161735Z 
2022-12-01T11:19:51.7164641Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test
2022-12-01T11:19:51.7166139Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_y7ib0qtl)
2022-12-01T11:19:51.7166591Z 
2022-12-01T11:19:51.7166736Z Running tests...
2022-12-01T11:19:51.7167261Z ----------------------------------------------------------------------
2022-12-01T11:19:51.7168888Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-12-01T11:19:51.7169384Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.553s)
2022-12-01T11:19:51.7169728Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.003s)
2022-12-01T11:19:51.7170151Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.003s)
2022-12-01T11:19:51.7170624Z   test_binary (__main__.StartProcessesListTest) ... hello stderr from 0
2022-12-01T11:19:51.7172536Z hello stdout from 0
2022-12-01T11:19:51.7172947Z hello stderr from 1
2022-12-01T11:19:51.7173173Z hello stdout from 1
2022-12-01T11:19:51.7173461Z ok (0.105s)
2022-12-01T11:19:51.7173793Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1
2022-12-01T11:19:51.7174120Z bar stdout from 1
2022-12-01T11:19:51.7179416Z failed (exitcode: 138) local_rank: 0 (pid: 74493) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-12-01T11:19:51.7179925Z ok (0.106s)
2022-12-01T11:19:51.7180291Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.003s)
2022-12-01T11:19:51.7180701Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-12-01T11:19:51.7181227Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-12-01T11:19:51.7181611Z     raise RuntimeError(f"raised from {rank}")
2022-12-01T11:19:51.7181934Z RuntimeError: raised from 0
2022-12-01T11:19:51.7182167Z bar from 1
2022-12-01T11:19:51.7182563Z failed (exitcode: 1) local_rank: 0 (pid: 74496) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-12-01T11:19:51.7183022Z ok (0.105s)
2022-12-01T11:19:51.7185711Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-12-01T11:19:51.7186241Z [trainer1]:world stderr from 1
2022-12-01T11:19:51.7186530Z [trainer0]:hello stdout from 0
2022-12-01T11:19:51.7186882Z ok (1.108s)
2022-12-01T11:19:51.7187589Z   test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7188115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7188731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7189203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7189782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7190224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7190780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7191232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7191561Z hello stderr from 0
2022-12-01T11:19:51.7191813Z hello stderr from 1
2022-12-01T11:19:51.7192044Z hello stdout from 1
2022-12-01T11:19:51.7192288Z hello stdout from 0
2022-12-01T11:19:51.7192663Z Closing process 74504 via signal SIGTERM
2022-12-01T11:19:51.7192937Z ok (1.841s)
2022-12-01T11:19:51.7193585Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7194115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7194728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7195173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7195739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7196178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7196749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7197213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7197789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7198213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7198784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7199246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7199813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7200266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7200842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7201311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7201642Z Closing process 74569 via signal SIGTERM
2022-12-01T11:19:51.7201949Z Closing process 74570 via signal SIGTERM
2022-12-01T11:19:51.7202250Z Closing process 74571 via signal SIGTERM
2022-12-01T11:19:51.7202496Z ok (2.400s)
2022-12-01T11:19:51.7202799Z   test_function_raise (__main__.StartProcessesListTest)
2022-12-01T11:19:51.7203569Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7204073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7204659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7205197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7205781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7206208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7206782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7207246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7207639Z failed (exitcode: 1) local_rank: 0 (pid: 74701) of fn: echo2 (start_method: spawn)
2022-12-01T11:19:51.7207996Z Traceback (most recent call last):
2022-12-01T11:19:51.7208540Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T11:19:51.7208983Z     self._pc.join(-1)
2022-12-01T11:19:51.7209438Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-12-01T11:19:51.7209902Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-12-01T11:19:51.7210361Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-12-01T11:19:51.7210635Z 
2022-12-01T11:19:51.7210848Z -- Process 0 terminated with the following error:
2022-12-01T11:19:51.7211175Z Traceback (most recent call last):
2022-12-01T11:19:51.7211671Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-12-01T11:19:51.7212026Z     fn(i, *args)
2022-12-01T11:19:51.7212511Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-12-01T11:19:51.7212910Z     ret = record(fn)(*args_)
2022-12-01T11:19:51.7213455Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-12-01T11:19:51.7213854Z     return f(*args, **kwargs)
2022-12-01T11:19:51.7214255Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-12-01T11:19:51.7215219Z     raise RuntimeError(msg)
2022-12-01T11:19:51.7215443Z RuntimeError: hello
2022-12-01T11:19:51.7215610Z 
2022-12-01T11:19:51.7215717Z ok (1.823s)
2022-12-01T11:19:51.7216055Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.003s)
2022-12-01T11:19:51.7216480Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.002s)
2022-12-01T11:19:51.7216935Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 74767 via signal SIGTERM
2022-12-01T11:19:51.7217305Z ok (0.005s)
2022-12-01T11:19:51.7217935Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-12-01T11:19:51.7218389Z Traceback (most recent call last):
2022-12-01T11:19:51.7218937Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T11:19:51.7219368Z     self._pc.join(-1)
2022-12-01T11:19:51.7219702Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-12-01T11:19:51.7220038Z     return self._mock_call(*args, **kwargs)
2022-12-01T11:19:51.7220393Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-12-01T11:19:51.7220760Z     return self._execute_mock_call(*args, **kwargs)
2022-12-01T11:19:51.7221121Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-12-01T11:19:51.7221450Z     raise effect
2022-12-01T11:19:51.7221824Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-12-01T11:19:51.7222270Z ok (0.005s)
2022-12-01T11:19:51.7222929Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7223533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7224118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7224573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7224888Z ok (2.813s)
2022-12-01T11:19:51.7225282Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 74801 closing signal SIGTERM
2022-12-01T11:19:51.7225638Z ok (0.005s)
2022-12-01T11:19:51.7225948Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.003s)
2022-12-01T11:19:51.7226351Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.002s)
2022-12-01T11:19:51.7227071Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7227579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7228157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7228632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7229191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7229642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7230218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7230685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7230974Z world
2022-12-01T11:19:51.7231188Z hello
2022-12-01T11:19:51.7231447Z Closing process 74803 via signal SIGTERM
2022-12-01T11:19:51.7231700Z ok (1.943s)
2022-12-01T11:19:51.7232024Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.003s)
2022-12-01T11:19:51.7232430Z   test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1
2022-12-01T11:19:51.7232729Z bar stdout from 1
2022-12-01T11:19:51.7233161Z failed (exitcode: 138) local_rank: 0 (pid: 74868) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-12-01T11:19:51.7233553Z ok (0.105s)
2022-12-01T11:19:51.7233890Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.003s)
2022-12-01T11:19:51.7234316Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-12-01T11:19:51.7234799Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-12-01T11:19:51.7235218Z     raise RuntimeError(f"raised from {rank}")
2022-12-01T11:19:51.7235505Z RuntimeError: raised from 0
2022-12-01T11:19:51.7235763Z bar from 1
2022-12-01T11:19:51.7236181Z failed (exitcode: 1) local_rank: 0 (pid: 74871) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-12-01T11:19:51.7236556Z ok (0.104s)
2022-12-01T11:19:51.7237194Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7237728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7238307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7238758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7239403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7239866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7240447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7240956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7241537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7241987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7242544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7243010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7243585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7244036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7244587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7245054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7245401Z Closing process 74873 via signal SIGTERM
2022-12-01T11:19:51.7245691Z Closing process 74874 via signal SIGTERM
2022-12-01T11:19:51.7245995Z Closing process 74876 via signal SIGTERM
2022-12-01T11:19:51.7246261Z ok (2.290s)
2022-12-01T11:19:51.7246536Z   test_function_raise (__main__.StartProcessesTest)
2022-12-01T11:19:51.7247213Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7247722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7248299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7248751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7249330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7249774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7250346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7250791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7251197Z failed (exitcode: 1) local_rank: 0 (pid: 75005) of fn: echo2 (start_method: spawn)
2022-12-01T11:19:51.7251549Z Traceback (most recent call last):
2022-12-01T11:19:51.7252114Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T11:19:51.7252554Z     self._pc.join(-1)
2022-12-01T11:19:51.7253028Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-12-01T11:19:51.7253477Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-12-01T11:19:51.7253933Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-12-01T11:19:51.7254201Z 
2022-12-01T11:19:51.7254666Z -- Process 0 terminated with the following error:
2022-12-01T11:19:51.7255021Z Traceback (most recent call last):
2022-12-01T11:19:51.7255480Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-12-01T11:19:51.7255843Z     fn(i, *args)
2022-12-01T11:19:51.7256350Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-12-01T11:19:51.7256724Z     ret = record(fn)(*args_)
2022-12-01T11:19:51.7257377Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-12-01T11:19:51.7257809Z     return f(*args, **kwargs)
2022-12-01T11:19:51.7258208Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-12-01T11:19:51.7258719Z     raise RuntimeError(msg)
2022-12-01T11:19:51.7258989Z RuntimeError: hello
2022-12-01T11:19:51.7259152Z 
2022-12-01T11:19:51.7259254Z ok (1.821s)
2022-12-01T11:19:51.7259561Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.003s)
2022-12-01T11:19:51.7259964Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.003s)
2022-12-01T11:19:51.7260417Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 75071 via signal SIGTERM
2022-12-01T11:19:51.7260756Z ok (0.004s)
2022-12-01T11:19:51.7261361Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-12-01T11:19:51.7261832Z Traceback (most recent call last):
2022-12-01T11:19:51.7262374Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-12-01T11:19:51.7262792Z     self._pc.join(-1)
2022-12-01T11:19:51.7263126Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-12-01T11:19:51.7263478Z     return self._mock_call(*args, **kwargs)
2022-12-01T11:19:51.7263816Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-12-01T11:19:51.7264180Z     return self._execute_mock_call(*args, **kwargs)
2022-12-01T11:19:51.7264559Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-12-01T11:19:51.7264886Z     raise effect
2022-12-01T11:19:51.7265241Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-12-01T11:19:51.7265597Z ok (0.005s)
2022-12-01T11:19:51.7266215Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7266706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7267289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7267758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7268052Z ok (2.867s)
2022-12-01T11:19:51.7268431Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 75105 closing signal SIGTERM
2022-12-01T11:19:51.7268792Z ok (0.004s)
2022-12-01T11:19:51.7269088Z   test_to_map (__main__.StartProcessesTest) ... ok (0.002s)
2022-12-01T11:19:51.7269449Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.002s)
2022-12-01T11:19:51.7270141Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7270655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7271217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7271693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7272270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:19:51.7272722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:19:51.7273276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:19:51.7273743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:19:51.7274044Z hello
2022-12-01T11:19:51.7274235Z world
2022-12-01T11:19:51.7274491Z Closing process 75106 via signal SIGTERM
2022-12-01T11:19:51.7274760Z ok (1.844s)
2022-12-01T11:19:51.7275098Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2022-12-01T11:19:51.7275448Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-12-01T11:19:51.7275864Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-12-01T11:19:51.7276065Z 
2022-12-01T11:19:51.7276342Z ----------------------------------------------------------------------
2022-12-01T11:19:51.7276657Z Ran 38 tests in 22.900s
2022-12-01T11:19:51.7276820Z 
2022-12-01T11:19:51.7276916Z OK
2022-12-01T11:19:51.7277050Z 
2022-12-01T11:19:51.7277174Z Generating XML reports...
2022-12-01T11:19:51.7277805Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221201111928.xml
2022-12-01T11:19:51.7278683Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221201111928.xml
2022-12-01T11:19:51.7279546Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221201111928.xml
2022-12-01T11:19:51.7280356Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221201111928.xml
2022-12-01T11:19:51.7280716Z 
2022-12-01T11:19:51.7281158Z ##[endgroup]
2022-12-01T11:19:51.7281832Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_y7ib0qtl)
2022-12-01T11:19:51.7282244Z 
2022-12-01T11:19:51.7282555Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-12-01 11:19:51.716368]
2022-12-01T11:19:51.7283290Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:19:51.716634]
2022-12-01T11:20:04.7226034Z 
2022-12-01T11:20:04.7226744Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan
2022-12-01T11:20:04.7227919Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_q8kly0ot)
2022-12-01T11:20:04.7228348Z 
2022-12-01T11:20:04.7228463Z Running tests...
2022-12-01T11:20:04.7228987Z ----------------------------------------------------------------------
2022-12-01T11:20:04.7229779Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan
2022-12-01T11:20:04.7230662Z   test_custom_sharding_planner (__main__.TestShardingPlan) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:20:04.7231125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75207
2022-12-01T11:20:04.7231547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75208
2022-12-01T11:20:04.7232031Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75209
2022-12-01T11:20:04.7232418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75210
2022-12-01T11:20:04.7233080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7233901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7234569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7235011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7235607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7235995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7236548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7237269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7237878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7238428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7238990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7239455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7240032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7240459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7241029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7241494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7241938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:04.7242399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:04.7242871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:04.7243335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:04.7243710Z skip: Need at least 4 CUDA devices (3.524s)
2022-12-01T11:20:04.7244197Z   test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75343
2022-12-01T11:20:04.7244737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75344
2022-12-01T11:20:04.7245182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75345
2022-12-01T11:20:04.7245606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75346
2022-12-01T11:20:04.7246210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7246667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7247221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7247694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7248273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7248718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7249270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7249734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7250315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7250756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7251316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7251769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7252343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7252764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7253335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7253848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7254708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:04.7255206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:04.7255691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:04.7256183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:04.7256534Z skip: Need at least 4 CUDA devices (1.910s)
2022-12-01T11:20:04.7257027Z   test_shard_module_sub_process_group (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75479
2022-12-01T11:20:04.7257560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75480
2022-12-01T11:20:04.7258006Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75481
2022-12-01T11:20:04.7258428Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75482
2022-12-01T11:20:04.7259047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7259501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7260080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7260530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7261133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7261549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7262100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7262563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7263175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7263643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7264266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7264606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7265182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7265640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7266174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7266632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7267066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:04.7267523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:04.7267986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:04.7268468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:04.7268829Z skip: Need at least 4 CUDA devices (1.910s)
2022-12-01T11:20:04.7269300Z   test_sharding_plan_errors (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75615
2022-12-01T11:20:04.7269868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75616
2022-12-01T11:20:04.7270269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75617
2022-12-01T11:20:04.7270693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75618
2022-12-01T11:20:04.7271370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7271829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7272406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7272918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7273513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7273937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7274487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7274952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7275525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7275988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7276520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7276984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7277557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7277977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7278544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7279005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7279466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:04.7279898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:04.7280355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:04.7280835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:04.7281213Z skip: Need at least 4 CUDA devices (1.909s)
2022-12-01T11:20:04.7281683Z   test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75751
2022-12-01T11:20:04.7282216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75752
2022-12-01T11:20:04.7282665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75753
2022-12-01T11:20:04.7283116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75754
2022-12-01T11:20:04.7283690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7284139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7284713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7285168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7285743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7286185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7286735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7287201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7287773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7288266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7288825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7289363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7289928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:04.7290366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:04.7290914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:04.7291375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:04.7291809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:04.7292261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:04.7292722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:04.7293187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:04.7293578Z skip: Need at least 4 CUDA devices (1.912s)
2022-12-01T11:20:04.7293753Z 
2022-12-01T11:20:04.7294026Z ----------------------------------------------------------------------
2022-12-01T11:20:04.7294623Z Ran 5 tests in 11.165s
2022-12-01T11:20:04.7294798Z 
2022-12-01T11:20:04.7294907Z OK (skipped=5)
2022-12-01T11:20:04.7295058Z 
2022-12-01T11:20:04.7295166Z Generating XML reports...
2022-12-01T11:20:04.7295750Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221201111953.xml
2022-12-01T11:20:04.7296118Z 
2022-12-01T11:20:04.7296447Z ##[endgroup]
2022-12-01T11:20:04.7297097Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_q8kly0ot)
2022-12-01T11:20:04.7297498Z 
2022-12-01T11:20:04.7297810Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2022-12-01 11:20:04.722818]
2022-12-01T11:20:04.7298562Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:20:04.723213]
2022-12-01T11:20:15.5872935Z 
2022-12-01T11:20:15.5873763Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_binary_cmp
2022-12-01T11:20:15.5875710Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_emulxltl)
2022-12-01T11:20:15.5876501Z 
2022-12-01T11:20:15.5876633Z Running tests...
2022-12-01T11:20:15.5877469Z ----------------------------------------------------------------------
2022-12-01T11:20:15.5878225Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2022-12-01T11:20:15.5878717Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2022-12-01T11:20:15.5879186Z Test torch.allclose(ShardedTensor, ShardedTensor) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:20:15.5879648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75922
2022-12-01T11:20:15.5880103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75923
2022-12-01T11:20:15.5880544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75924
2022-12-01T11:20:15.5881336Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75925
2022-12-01T11:20:15.5882095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5882537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5883356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5883858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5884547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5884998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5885561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5886010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5886564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5887035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5887622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5888067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5888629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5889091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5889678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5890128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5890571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:15.5891044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:15.5891517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:15.5891969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:15.5892354Z skip: Need at least 4 CUDA devices (3.478s)
2022-12-01T11:20:15.5892869Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76058
2022-12-01T11:20:15.5893409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76059
2022-12-01T11:20:15.5893858Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76060
2022-12-01T11:20:15.5894295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76061
2022-12-01T11:20:15.5895190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5895624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5896195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5896639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5897216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5897672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5898254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5898718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5899275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5899719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5900379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5900861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5901424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5901953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5902524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5902971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5903411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:15.5903882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:15.5904345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:15.5904793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:15.5905178Z skip: Need at least 4 CUDA devices (1.809s)
2022-12-01T11:20:15.5905538Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2022-12-01T11:20:15.5906024Z Test torch.equal(ShardedTensor, ShardedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76194
2022-12-01T11:20:15.5906541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76195
2022-12-01T11:20:15.5906985Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76196
2022-12-01T11:20:15.5907423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76197
2022-12-01T11:20:15.5908021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5908474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5909055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5909526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5910093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5910536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5911105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5911556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5912133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5912572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5913147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5913594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5914178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5914618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5915169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5915632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5916072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:15.5916544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:15.5916990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:15.5917503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:15.5917914Z skip: Need at least 4 CUDA devices (1.808s)
2022-12-01T11:20:15.5918451Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76330
2022-12-01T11:20:15.5919005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76331
2022-12-01T11:20:15.5919451Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76332
2022-12-01T11:20:15.5919889Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76333
2022-12-01T11:20:15.5920482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5920930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5921511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5921979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5922545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5922987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5923556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5924003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5924572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5925013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5925580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5926033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5926607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:15.5927054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:15.5927624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:15.5928072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:15.5928510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:15.5928981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:15.5929433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:15.5929892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:15.5930284Z skip: Need at least 4 CUDA devices (1.947s)
2022-12-01T11:20:15.5930480Z 
2022-12-01T11:20:15.5930752Z ----------------------------------------------------------------------
2022-12-01T11:20:15.5931070Z Ran 4 tests in 9.042s
2022-12-01T11:20:15.5931233Z 
2022-12-01T11:20:15.5931342Z OK (skipped=4)
2022-12-01T11:20:15.5931495Z 
2022-12-01T11:20:15.5931623Z Generating XML reports...
2022-12-01T11:20:15.5932277Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221201112006.xml
2022-12-01T11:20:15.5932678Z 
2022-12-01T11:20:15.5933008Z ##[endgroup]
2022-12-01T11:20:15.5933684Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_emulxltl)
2022-12-01T11:20:15.5934084Z 
2022-12-01T11:20:15.5934716Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2022-12-01 11:20:15.587505]
2022-12-01T11:20:15.5935513Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:20:15.587871]
2022-12-01T11:20:26.5479434Z 
2022-12-01T11:20:26.5480277Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2022-12-01T11:20:26.5481540Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_ue88rp61)
2022-12-01T11:20:26.5481921Z 
2022-12-01T11:20:26.5482056Z Running tests...
2022-12-01T11:20:26.5482600Z ----------------------------------------------------------------------
2022-12-01T11:20:26.5483185Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2022-12-01T11:20:26.5483792Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:20:26.5484341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76501
2022-12-01T11:20:26.5484745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76502
2022-12-01T11:20:26.5485228Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76503
2022-12-01T11:20:26.5485668Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76504
2022-12-01T11:20:26.5486303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5486714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5487300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5487833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5488356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5488789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5489327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5489791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5490348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5490792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5491360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5491805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5492386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5492827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5493398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5493843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5494279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:26.5495022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:26.5495511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:26.5495935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:26.5496320Z skip: Need at least 4 CUDA devices (3.512s)
2022-12-01T11:20:26.5497063Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76637
2022-12-01T11:20:26.5497766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76638
2022-12-01T11:20:26.5498193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76639
2022-12-01T11:20:26.5498633Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76640
2022-12-01T11:20:26.5499251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5499708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5500249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5500695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5501271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5501725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5502305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5502747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5503322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5503762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5504344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5504806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5505384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5505812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5506384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5506846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5507266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:26.5507736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:26.5508202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:26.5508667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:26.5509039Z skip: Need at least 4 CUDA devices (1.809s)
2022-12-01T11:20:26.5509543Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76773
2022-12-01T11:20:26.5510092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76774
2022-12-01T11:20:26.5510526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76775
2022-12-01T11:20:26.5510967Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76776
2022-12-01T11:20:26.5511577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5512019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5512571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5513034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5513672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5514123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5514735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5515199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5515770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5516191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5516758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5517219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5517794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5518211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5518774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5519235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5519652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:26.5520120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:26.5520577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:26.5521040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:26.5521409Z skip: Need at least 4 CUDA devices (1.909s)
2022-12-01T11:20:26.5521923Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76909
2022-12-01T11:20:26.5522484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76910
2022-12-01T11:20:26.5522933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76911
2022-12-01T11:20:26.5523368Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76912
2022-12-01T11:20:26.5523960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5524410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5524963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5525428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5525998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5526437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5526991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5527429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5527998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5528444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5529023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5529479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5530050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:26.5530530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:26.5531113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:26.5531630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:26.5532045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:26.5532515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:26.5532997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:26.5533437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:26.5533809Z skip: Need at least 4 CUDA devices (1.909s)
2022-12-01T11:20:26.5534001Z 
2022-12-01T11:20:26.5534280Z ----------------------------------------------------------------------
2022-12-01T11:20:26.5534794Z Ran 4 tests in 9.140s
2022-12-01T11:20:26.5534959Z 
2022-12-01T11:20:26.5535067Z OK (skipped=4)
2022-12-01T11:20:26.5535204Z 
2022-12-01T11:20:26.5535327Z Generating XML reports...
2022-12-01T11:20:26.5536034Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221201112017.xml
2022-12-01T11:20:26.5536473Z 
2022-12-01T11:20:26.5536784Z ##[endgroup]
2022-12-01T11:20:26.5537467Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_ue88rp61)
2022-12-01T11:20:26.5537885Z 
2022-12-01T11:20:26.5538185Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2022-12-01 11:20:26.548013]
2022-12-01T11:20:26.5538908Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:20:26.548310]
2022-12-01T11:20:35.5951522Z 
2022-12-01T11:20:35.5952216Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_init
2022-12-01T11:20:35.5953220Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_mi7pxz84)
2022-12-01T11:20:35.5953793Z 
2022-12-01T11:20:35.5953965Z Running tests...
2022-12-01T11:20:35.5954517Z ----------------------------------------------------------------------
2022-12-01T11:20:35.5955358Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init
2022-12-01T11:20:35.5955933Z   test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit)
2022-12-01T11:20:35.5956680Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:20:35.5957157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77080
2022-12-01T11:20:35.5957890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77081
2022-12-01T11:20:35.5958325Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77082
2022-12-01T11:20:35.5958833Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77083
2022-12-01T11:20:35.5959660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5960153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5961230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5962095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5963316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5964167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5964771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5965342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5965928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5966354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5966987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5967461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5968039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5968494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5969055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5969533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5969975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:35.5970429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:35.5970904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:35.5971366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:35.5971761Z skip: Need at least 4 CUDA devices (3.466s)
2022-12-01T11:20:35.5972125Z   test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit)
2022-12-01T11:20:35.5972660Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77216
2022-12-01T11:20:35.5973186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77217
2022-12-01T11:20:35.5973638Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77218
2022-12-01T11:20:35.5974071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77219
2022-12-01T11:20:35.5975123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5975580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5976134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5976586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5977161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5977644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5978216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5978693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5979273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5979699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5980274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5980741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5981320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5981849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5982438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5982977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5983421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:35.5983878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:35.5984337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:35.5984808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:35.5985186Z skip: Need at least 4 CUDA devices (1.809s)
2022-12-01T11:20:35.5985573Z   test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit)
2022-12-01T11:20:35.5986104Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77352
2022-12-01T11:20:35.5986625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77353
2022-12-01T11:20:35.5987061Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77354
2022-12-01T11:20:35.5987499Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77355
2022-12-01T11:20:35.5988119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5988553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5989132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5989601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5990184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5990613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5991186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5991658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5992216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5992663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5993231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5993694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5994251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:35.5994704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:35.5995277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:35.5995744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:35.5996162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:35.5996637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:35.5997105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:35.5997555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:35.5997941Z skip: Need at least 4 CUDA devices (1.909s)
2022-12-01T11:20:35.5998136Z 
2022-12-01T11:20:35.5998414Z ----------------------------------------------------------------------
2022-12-01T11:20:35.5998810Z Ran 3 tests in 7.185s
2022-12-01T11:20:35.5998964Z 
2022-12-01T11:20:35.5999075Z OK (skipped=3)
2022-12-01T11:20:35.5999232Z 
2022-12-01T11:20:35.5999360Z Generating XML reports...
2022-12-01T11:20:35.6000060Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221201112028.xml
2022-12-01T11:20:35.6000447Z 
2022-12-01T11:20:35.6000764Z ##[endgroup]
2022-12-01T11:20:35.6001420Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_mi7pxz84)
2022-12-01T11:20:35.6001808Z 
2022-12-01T11:20:35.6002109Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-12-01 11:20:35.595242]
2022-12-01T11:20:35.6002837Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:20:35.595518]
2022-12-01T11:20:42.7591174Z 
2022-12-01T11:20:42.7591904Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_chunk
2022-12-01T11:20:42.7593120Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk__vz84vw8)
2022-12-01T11:20:42.7593510Z 
2022-12-01T11:20:42.7593626Z Running tests...
2022-12-01T11:20:42.7594126Z ----------------------------------------------------------------------
2022-12-01T11:20:42.7594700Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-12-01T11:20:42.7595261Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:20:42.7595734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77523
2022-12-01T11:20:42.7596186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77524
2022-12-01T11:20:42.7596628Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77525
2022-12-01T11:20:42.7597063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77526
2022-12-01T11:20:42.7597653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7598114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7598713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7599149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7599730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7600175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7600764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7601173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7601752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7602224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7602815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7603264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7603840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7604301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7605101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7605592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7606034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:42.7606616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:42.7607073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:42.7607531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:42.7607925Z skip: Need at least 4 CUDA devices (3.431s)
2022-12-01T11:20:42.7608397Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77659
2022-12-01T11:20:42.7608942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77660
2022-12-01T11:20:42.7609390Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77661
2022-12-01T11:20:42.7609836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77662
2022-12-01T11:20:42.7610451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7610894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7611494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7611969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7612531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7612976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7613550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7614018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7614868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7615326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7615865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7616311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7616888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:42.7617329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:42.7617898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:42.7618347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:42.7618786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:42.7619268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:42.7619718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:42.7620185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:42.7620571Z skip: Need at least 4 CUDA devices (1.909s)
2022-12-01T11:20:42.7620766Z 
2022-12-01T11:20:42.7621042Z ----------------------------------------------------------------------
2022-12-01T11:20:42.7621379Z Ran 2 tests in 5.340s
2022-12-01T11:20:42.7621520Z 
2022-12-01T11:20:42.7621628Z OK (skipped=2)
2022-12-01T11:20:42.7621780Z 
2022-12-01T11:20:42.7621905Z Generating XML reports...
2022-12-01T11:20:42.7622651Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221201112037.xml
2022-12-01T11:20:42.7623048Z 
2022-12-01T11:20:42.7623370Z ##[endgroup]
2022-12-01T11:20:42.7624108Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk__vz84vw8)
2022-12-01T11:20:42.7624497Z 
2022-12-01T11:20:42.7624806Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-12-01 11:20:42.759184]
2022-12-01T11:20:42.7625525Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:20:42.759497]
2022-12-01T11:20:50.0124562Z 
2022-12-01T11:20:50.0125100Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding
2022-12-01T11:20:50.0126918Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_nxbbbznc)
2022-12-01T11:20:50.0127858Z 
2022-12-01T11:20:50.0128057Z Running tests...
2022-12-01T11:20:50.0128572Z ----------------------------------------------------------------------
2022-12-01T11:20:50.0129182Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2022-12-01T11:20:50.0129722Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:20:50.0130219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77830
2022-12-01T11:20:50.0130636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77831
2022-12-01T11:20:50.0131083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77832
2022-12-01T11:20:50.0131541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77833
2022-12-01T11:20:50.0132167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0132608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0133182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0133631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0134190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0135007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0135588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0136058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0136644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0137072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0137638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0138086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0138640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0139115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0139689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0140134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0140847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:50.0141422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:50.0141931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:50.0142534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:50.0142951Z skip: Need at least 4 CUDA devices (3.372s)
2022-12-01T11:20:50.0143474Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77966
2022-12-01T11:20:50.0144062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77967
2022-12-01T11:20:50.0144510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77968
2022-12-01T11:20:50.0144988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77969
2022-12-01T11:20:50.0145653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0146139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0146739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0147234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0147849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0148304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0148931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0149411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0150049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0150483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0151089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0151593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0152189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:20:50.0152671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:20:50.0153269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:20:50.0153770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:20:50.0154202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:20:50.0154710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:20:50.0155209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:20:50.0155713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:20:50.0156109Z skip: Need at least 4 CUDA devices (2.009s)
2022-12-01T11:20:50.0156313Z 
2022-12-01T11:20:50.0156600Z ----------------------------------------------------------------------
2022-12-01T11:20:50.0156946Z Ran 2 tests in 5.382s
2022-12-01T11:20:50.0157114Z 
2022-12-01T11:20:50.0157208Z OK (skipped=2)
2022-12-01T11:20:50.0157370Z 
2022-12-01T11:20:50.0157498Z Generating XML reports...
2022-12-01T11:20:50.0158170Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221201112044.xml
2022-12-01T11:20:50.0158578Z 
2022-12-01T11:20:50.0158900Z ##[endgroup]
2022-12-01T11:20:50.0159756Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_nxbbbznc)
2022-12-01T11:20:50.0160258Z 
2022-12-01T11:20:50.0160547Z Running distributed/test_data_parallel ... [2022-12-01 11:20:50.012497]
2022-12-01T11:20:50.0161274Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:20:50.012757]
2022-12-01T11:20:58.5339063Z 
2022-12-01T11:20:58.5339710Z Expand the folded group to see the log file of distributed/test_data_parallel
2022-12-01T11:20:58.5340636Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel__24mue_d)
2022-12-01T11:20:58.5341018Z 
2022-12-01T11:20:58.5341142Z Running tests...
2022-12-01T11:20:58.5341671Z ----------------------------------------------------------------------
2022-12-01T11:20:58.5342248Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2022-12-01T11:20:58.5342683Z   test_autocast (__main__.TestDataParallel) ... ok (1.809s)
2022-12-01T11:20:58.5343044Z   test_data_parallel (__main__.TestDataParallel) ... ok (0.176s)
2022-12-01T11:20:58.5343469Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.012s)
2022-12-01T11:20:58.5343881Z   test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.003s)
2022-12-01T11:20:58.5344302Z   test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.006s)
2022-12-01T11:20:58.5344751Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.006s)
2022-12-01T11:20:58.5345708Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-12-01T11:20:58.5346406Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-12-01T11:20:58.5346779Z ok (0.002s)
2022-12-01T11:20:58.5347096Z   test_data_parallel_model_device (__main__.TestDataParallel)
2022-12-01T11:20:58.5347458Z Test device[0] check at forward time. ... ok (0.034s)
2022-12-01T11:20:58.5347761Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.072s)
2022-12-01T11:20:58.5348199Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.005s)
2022-12-01T11:20:58.5349074Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/comm.py:231: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".
2022-12-01T11:20:58.5349637Z   warnings.warn(
2022-12-01T11:20:58.5349860Z ok (0.022s)
2022-12-01T11:20:58.5350192Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.003s)
2022-12-01T11:20:58.5350620Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.006s)
2022-12-01T11:20:58.5351018Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.004s)
2022-12-01T11:20:58.5351419Z   test_data_parallel_rnn (__main__.TestDataParallel) ... ok (1.042s)
2022-12-01T11:20:58.5351821Z   test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.004s)
2022-12-01T11:20:58.5352208Z   test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.012s)
2022-12-01T11:20:58.5353068Z   test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
2022-12-01T11:20:58.5353764Z   warnings.warn('Was asked to gather along dimension 0, but all '
2022-12-01T11:20:58.5354068Z ok (0.045s)
2022-12-01T11:20:58.5354625Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s)
2022-12-01T11:20:58.5355045Z   test_gather_gpu (__main__.TestDataParallel) ... ok (0.044s)
2022-12-01T11:20:58.5355423Z   test_parallel_apply (__main__.TestDataParallel) ... ok (0.005s)
2022-12-01T11:20:58.5355921Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.006s)
2022-12-01T11:20:58.5356356Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.002s)
2022-12-01T11:20:58.5356789Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.007s)
2022-12-01T11:20:58.5357186Z   test_replicate (__main__.TestDataParallel) ... ok (0.004s)
2022-12-01T11:20:58.5357572Z   test_replicate_buffers (__main__.TestDataParallel) ... ok (0.003s)
2022-12-01T11:20:58.5357948Z   test_save_replica_module (__main__.TestDataParallel) ... ok (0.003s)
2022-12-01T11:20:58.5358330Z   test_scatter_cpu (__main__.TestDataParallel) ... ok (0.018s)
2022-12-01T11:20:58.5358697Z   test_scatter_gpu (__main__.TestDataParallel) ... ok (0.019s)
2022-12-01T11:20:58.5359065Z   test_strided_grad_layout (__main__.TestDataParallel) ... ok (1.215s)
2022-12-01T11:20:58.5359439Z   test_zero_grad (__main__.TestDataParallel) ... ok (0.007s)
2022-12-01T11:20:58.5359879Z   test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5360427Z   test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5360940Z   test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5361469Z   test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5362002Z   test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5362519Z   test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5363069Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5363635Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5364192Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5364735Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5365283Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5365835Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5366393Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-12-01T11:20:58.5366940Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5367500Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-12-01T11:20:58.5367803Z 
2022-12-01T11:20:58.5368090Z ----------------------------------------------------------------------
2022-12-01T11:20:58.5368426Z Ran 46 tests in 4.669s
2022-12-01T11:20:58.5368572Z 
2022-12-01T11:20:58.5368664Z OK
2022-12-01T11:20:58.5368796Z 
2022-12-01T11:20:58.5368919Z Generating XML reports...
2022-12-01T11:20:58.5369508Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221201112053.xml
2022-12-01T11:20:58.5370290Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20221201112053.xml
2022-12-01T11:20:58.5370678Z 
2022-12-01T11:20:58.5370996Z ##[endgroup]
2022-12-01T11:20:58.5371651Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel__24mue_d)
2022-12-01T11:20:58.5372106Z 
2022-12-01T11:20:58.5372422Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2022-12-01 11:20:58.534041]
2022-12-01T11:20:58.5373143Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:20:58.534327]
2022-12-01T11:21:03.9073961Z 
2022-12-01T11:21:03.9074672Z Expand the folded group to see the log file of distributed/_shard/sharded_optim/test_sharded_optim
2022-12-01T11:21:03.9075711Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_769lyzg9)
2022-12-01T11:21:03.9076163Z 
2022-12-01T11:21:03.9076203Z Running tests...
2022-12-01T11:21:03.9076742Z ----------------------------------------------------------------------
2022-12-01T11:21:03.9077348Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2022-12-01T11:21:03.9078023Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:21:03.9079192Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82023 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2022-12-01T11:21:03.9080007Z   test_sharded_optim (__main__.TestShardedOptimizer) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78348
2022-12-01T11:21:03.9080539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78349
2022-12-01T11:21:03.9081012Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78350
2022-12-01T11:21:03.9081431Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78351
2022-12-01T11:21:03.9082037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:03.9082495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:03.9083059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:03.9083552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:03.9084136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:03.9084588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:03.9085141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:03.9085611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:03.9086190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:03.9086645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:03.9087202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:03.9087664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:03.9088239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:03.9088665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:03.9089239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:03.9089954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:03.9090419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:21:03.9090978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:21:03.9091445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:21:03.9091912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:21:03.9092289Z skip: Need at least 4 CUDA devices (1.912s)
2022-12-01T11:21:03.9092484Z 
2022-12-01T11:21:03.9092761Z ----------------------------------------------------------------------
2022-12-01T11:21:03.9093095Z Ran 2 tests in 3.519s
2022-12-01T11:21:03.9093261Z 
2022-12-01T11:21:03.9093373Z OK (skipped=2)
2022-12-01T11:21:03.9093508Z 
2022-12-01T11:21:03.9093635Z Generating XML reports...
2022-12-01T11:21:03.9094278Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221201112100.xml
2022-12-01T11:21:03.9094929Z 
2022-12-01T11:21:03.9095250Z ##[endgroup]
2022-12-01T11:21:03.9095917Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_769lyzg9)
2022-12-01T11:21:03.9096315Z 
2022-12-01T11:21:03.9096641Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-12-01 11:21:03.907353]
2022-12-01T11:21:03.9097402Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:21:03.907741]
2022-12-01T11:21:09.2589746Z 
2022-12-01T11:21:09.2590268Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2022-12-01T11:21:09.2591618Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_qw7rr4on)
2022-12-01T11:21:09.2592055Z 
2022-12-01T11:21:09.2592164Z Running tests...
2022-12-01T11:21:09.2592687Z ----------------------------------------------------------------------
2022-12-01T11:21:09.2593310Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-12-01T11:21:09.2593901Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:21:09.2594410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78519
2022-12-01T11:21:09.2594863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78520
2022-12-01T11:21:09.2595309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78521
2022-12-01T11:21:09.2595756Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78522
2022-12-01T11:21:09.2596375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:09.2596840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:09.2597417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:09.2597875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:09.2598454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:09.2598904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:09.2599476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:09.2599926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:09.2600764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:09.2601272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:09.2601986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:09.2602486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:09.2603105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:21:09.2603581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:21:09.2604169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:21:09.2604690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:21:09.2605163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-12-01T11:21:09.2605667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:21:09.2606152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-12-01T11:21:09.2606645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:21:09.2607059Z skip: Need at least 4 CUDA devices (3.523s)
2022-12-01T11:21:09.2607266Z 
2022-12-01T11:21:09.2607541Z ----------------------------------------------------------------------
2022-12-01T11:21:09.2607885Z Ran 1 test in 3.524s
2022-12-01T11:21:09.2608054Z 
2022-12-01T11:21:09.2608165Z OK (skipped=1)
2022-12-01T11:21:09.2608327Z 
2022-12-01T11:21:09.2608456Z Generating XML reports...
2022-12-01T11:21:09.2609178Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221201112105.xml
2022-12-01T11:21:09.2609634Z 
2022-12-01T11:21:09.2609950Z ##[endgroup]
2022-12-01T11:21:09.2610690Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_qw7rr4on)
2022-12-01T11:21:09.2611148Z 
2022-12-01T11:21:09.2611427Z Running distributed/elastic/utils/logging_test ... [2022-12-01 11:21:09.258989]
2022-12-01T11:21:09.2612162Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:21:09.259356]
2022-12-01T11:21:12.7402443Z 
2022-12-01T11:21:12.7403031Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test
2022-12-01T11:21:12.7404745Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_pr630n7r)
2022-12-01T11:21:12.7405518Z 
2022-12-01T11:21:12.7405720Z Running tests...
2022-12-01T11:21:12.7406268Z ----------------------------------------------------------------------
2022-12-01T11:21:12.7406833Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-12-01T11:21:12.7407313Z   test_derive_module_name (__main__.LoggingTest) ... ok (1.603s)
2022-12-01T11:21:12.7407611Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2022-12-01T11:21:12.7407820Z 
2022-12-01T11:21:12.7408087Z ----------------------------------------------------------------------
2022-12-01T11:21:12.7408484Z Ran 2 tests in 1.606s
2022-12-01T11:21:12.7408652Z 
2022-12-01T11:21:12.7408724Z OK
2022-12-01T11:21:12.7408856Z 
2022-12-01T11:21:12.7408986Z Generating XML reports...
2022-12-01T11:21:12.7409546Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221201112110.xml
2022-12-01T11:21:12.7409972Z 
2022-12-01T11:21:12.7410198Z ##[endgroup]
2022-12-01T11:21:12.7411092Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_pr630n7r)
2022-12-01T11:21:12.7411617Z 
2022-12-01T11:21:12.7411917Z Running distributed/elastic/metrics/api_test ... [2022-12-01 11:21:12.740279]
2022-12-01T11:21:12.7412656Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:21:12.740544]
2022-12-01T11:21:16.0895675Z 
2022-12-01T11:21:16.0896238Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test
2022-12-01T11:21:16.0897224Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_k_vq_14m)
2022-12-01T11:21:16.0897618Z 
2022-12-01T11:21:16.0897714Z Running tests...
2022-12-01T11:21:16.0898236Z ----------------------------------------------------------------------
2022-12-01T11:21:16.0898807Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2022-12-01T11:21:16.0899278Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (1.558s)
2022-12-01T11:21:16.0899648Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.002s)
2022-12-01T11:21:16.0899997Z   test_profile (__main__.MetricsApiTest) ... ok (0.002s)
2022-12-01T11:21:16.0900245Z 
2022-12-01T11:21:16.0900480Z ----------------------------------------------------------------------
2022-12-01T11:21:16.0900792Z Ran 3 tests in 1.561s
2022-12-01T11:21:16.0900956Z 
2022-12-01T11:21:16.0901051Z OK
2022-12-01T11:21:16.0901186Z 
2022-12-01T11:21:16.0901324Z Generating XML reports...
2022-12-01T11:21:16.0901945Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221201112114.xml
2022-12-01T11:21:16.0902281Z 
2022-12-01T11:21:16.0902588Z ##[endgroup]
2022-12-01T11:21:16.0903215Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_k_vq_14m)
2022-12-01T11:21:16.0903585Z 
2022-12-01T11:21:16.0903830Z Running distributed/test_nccl ... [2022-12-01 11:21:16.089594]
2022-12-01T11:21:16.0904459Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:21:16.089855]
2022-12-01T11:21:20.6857730Z 
2022-12-01T11:21:20.6858229Z Expand the folded group to see the log file of distributed/test_nccl
2022-12-01T11:21:20.6859352Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_wh34a19u)
2022-12-01T11:21:20.6859679Z 
2022-12-01T11:21:20.6859772Z Running tests...
2022-12-01T11:21:20.6860241Z ----------------------------------------------------------------------
2022-12-01T11:21:20.6860772Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2022-12-01T11:21:20.6861470Z   test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (1.155s)
2022-12-01T11:21:20.6861863Z   test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-12-01T11:21:20.6862217Z   test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-12-01T11:21:20.6862604Z   test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-12-01T11:21:20.6862921Z   test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-12-01T11:21:20.6863438Z   test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-12-01T11:21:20.6863824Z   test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.002s)
2022-12-01T11:21:20.6864123Z   test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-12-01T11:21:20.6864470Z   test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-12-01T11:21:20.6864856Z   test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-12-01T11:21:20.6865603Z   test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-12-01T11:21:20.6865918Z   test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.001s)
2022-12-01T11:21:20.6866128Z 
2022-12-01T11:21:20.6866519Z ----------------------------------------------------------------------
2022-12-01T11:21:20.6866898Z Ran 12 tests in 1.190s
2022-12-01T11:21:20.6867043Z 
2022-12-01T11:21:20.6867138Z OK
2022-12-01T11:21:20.6867253Z 
2022-12-01T11:21:20.6867380Z Generating XML reports...
2022-12-01T11:21:20.6867949Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221201112119.xml
2022-12-01T11:21:20.6868271Z 
2022-12-01T11:21:20.6868592Z ##[endgroup]
2022-12-01T11:21:20.6869128Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_wh34a19u)
2022-12-01T11:21:20.6869449Z 
2022-12-01T11:21:20.6869760Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2022-12-01 11:21:20.685794]
2022-12-01T11:21:20.6870510Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:21:20.686051]
2022-12-01T11:21:22.3274220Z 
2022-12-01T11:21:22.3274802Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_math_ops
2022-12-01T11:21:22.3276106Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_lxmfsy_3)
2022-12-01T11:21:22.3276522Z 
2022-12-01T11:21:22.3276822Z ##[endgroup]
2022-12-01T11:21:22.3277704Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_lxmfsy_3)
2022-12-01T11:21:22.3278168Z 
2022-12-01T11:21:22.3278472Z Running distributed/elastic/events/lib_test ... [2022-12-01 11:21:22.327443]
2022-12-01T11:21:22.3279652Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2022-12-01 11:21:22.327702]
2022-12-01T11:21:24.8858874Z 
2022-12-01T11:21:24.8859365Z Expand the folded group to see the log file of distributed/elastic/events/lib_test
2022-12-01T11:21:24.8860543Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_k5em6tat)
2022-12-01T11:21:24.8861072Z ============================= test session starts ==============================
2022-12-01T11:21:24.8861683Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:24.8862021Z cachedir: .pytest_cache
2022-12-01T11:21:24.8862785Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:24.8863306Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:24.8863921Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:24.8864295Z collecting ... collected 8 items
2022-12-01T11:21:24.8865558Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str
2022-12-01T11:21:24.8866661Z 
2022-12-01T11:21:24.8867108Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%]
2022-12-01T11:21:24.8867604Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%]
2022-12-01T11:21:24.8868167Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%]
2022-12-01T11:21:24.8868669Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%]
2022-12-01T11:21:24.8869215Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%]
2022-12-01T11:21:24.8869749Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%]
2022-12-01T11:21:24.8870246Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%]
2022-12-01T11:21:24.8870720Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%]
2022-12-01T11:21:24.8870982Z 
2022-12-01T11:21:24.8871160Z ============================== 8 passed in 1.36s ===============================
2022-12-01T11:21:24.8871354Z 
2022-12-01T11:21:24.8871694Z ##[endgroup]
2022-12-01T11:21:24.8872329Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_k5em6tat)
2022-12-01T11:21:24.8872694Z 
2022-12-01T11:21:24.8872970Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-12-01 11:21:24.885927]
2022-12-01T11:21:24.8873589Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-12-01 11:21:24.886185]
2022-12-01T11:21:32.3007269Z 
2022-12-01T11:21:32.3007839Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe
2022-12-01T11:21:32.3008978Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_39o3u8ru)
2022-12-01T11:21:32.3009786Z ============================= test session starts ==============================
2022-12-01T11:21:32.3010612Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:32.3010985Z cachedir: .pytest_cache
2022-12-01T11:21:32.3011601Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:32.3012394Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:21:32.3012749Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:32.3013317Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:32.3013691Z collecting ... collected 13 items
2022-12-01T11:21:32.3016190Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip
2022-12-01T11:21:32.3017517Z 
2022-12-01T11:21:32.3017824Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] PASSED  [  7%]
2022-12-01T11:21:32.3018357Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] PASSED [ 15%]
2022-12-01T11:21:32.3019123Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] PASSED [ 23%]
2022-12-01T11:21:32.3019684Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2022-12-01T11:21:32.3020339Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] PASSED [ 38%]
2022-12-01T11:21:32.3020866Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] PASSED [ 46%]
2022-12-01T11:21:32.3021373Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] PASSED [ 53%]
2022-12-01T11:21:32.3021908Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2022-12-01T11:21:32.3022454Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] PASSED [ 69%]
2022-12-01T11:21:32.3023001Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] PASSED [ 76%]
2022-12-01T11:21:32.3023523Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] PASSED [ 84%]
2022-12-01T11:21:32.3024073Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2022-12-01T11:21:32.3024526Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2022-12-01T11:21:32.3024776Z 
2022-12-01T11:21:32.3024928Z ======================== 10 passed, 3 skipped in 5.05s =========================
2022-12-01T11:21:32.3025135Z 
2022-12-01T11:21:32.3025457Z ##[endgroup]
2022-12-01T11:21:32.3026109Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_39o3u8ru)
2022-12-01T11:21:32.3026507Z 
2022-12-01T11:21:32.3026778Z Running distributed/pipeline/sync/skip/test_portal ... [2022-12-01 11:21:32.300720]
2022-12-01T11:21:32.3027399Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-12-01 11:21:32.300984]
2022-12-01T11:21:35.6833790Z 
2022-12-01T11:21:35.6834485Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal
2022-12-01T11:21:35.6835617Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_lz9m73oo)
2022-12-01T11:21:35.6836186Z ============================= test session starts ==============================
2022-12-01T11:21:35.6836846Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:35.6837143Z cachedir: .pytest_cache
2022-12-01T11:21:35.6837741Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:35.6838151Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:21:35.6838460Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:35.6839068Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:35.6839418Z collecting ... collected 10 items
2022-12-01T11:21:35.6841028Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1
2022-12-01T11:21:35.6842222Z 
2022-12-01T11:21:35.6842736Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%]
2022-12-01T11:21:35.6843264Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-12-01T11:21:35.6843883Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-12-01T11:21:35.6844357Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-12-01T11:21:35.6844851Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-12-01T11:21:35.6845393Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-12-01T11:21:35.6845924Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-12-01T11:21:35.6846411Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-12-01T11:21:35.6846927Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-12-01T11:21:35.6847469Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-12-01T11:21:35.6847767Z 
2022-12-01T11:21:35.6847936Z ============================== 10 passed in 1.39s ==============================
2022-12-01T11:21:35.6848123Z 
2022-12-01T11:21:35.6848444Z ##[endgroup]
2022-12-01T11:21:35.6849154Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_lz9m73oo)
2022-12-01T11:21:35.6849581Z 
2022-12-01T11:21:35.6849910Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-12-01 11:21:35.683349]
2022-12-01T11:21:35.6850594Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-12-01 11:21:35.683614]
2022-12-01T11:21:37.6491391Z 
2022-12-01T11:21:37.6492022Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables
2022-12-01T11:21:37.6493404Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_qgj7o3ob)
2022-12-01T11:21:37.6493998Z ============================= test session starts ==============================
2022-12-01T11:21:37.6495230Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:37.6495590Z cachedir: .pytest_cache
2022-12-01T11:21:37.6496281Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:37.6496731Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:21:37.6497054Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:37.6497674Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:37.6497983Z collecting ... collected 9 items
2022-12-01T11:21:37.6499337Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated
2022-12-01T11:21:37.6500507Z 
2022-12-01T11:21:37.6500742Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2022-12-01T11:21:37.6501464Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2022-12-01T11:21:37.6501966Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2022-12-01T11:21:37.6502544Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2022-12-01T11:21:37.6503014Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2022-12-01T11:21:37.6503503Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2022-12-01T11:21:37.6504032Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2022-12-01T11:21:37.6504535Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2022-12-01T11:21:37.6505022Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2022-12-01T11:21:37.6505310Z 
2022-12-01T11:21:37.6505474Z ============================== 9 passed in 0.05s ===============================
2022-12-01T11:21:37.6505670Z 
2022-12-01T11:21:37.6505988Z ##[endgroup]
2022-12-01T11:21:37.6506717Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_qgj7o3ob)
2022-12-01T11:21:37.6507131Z 
2022-12-01T11:21:37.6507424Z Running distributed/pipeline/sync/test_checkpoint ... [2022-12-01 11:21:37.649187]
2022-12-01T11:21:37.6508044Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-12-01 11:21:37.649520]
2022-12-01T11:21:41.0280891Z 
2022-12-01T11:21:41.0281460Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint
2022-12-01T11:21:41.0282729Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_rhjrix_z)
2022-12-01T11:21:41.0283282Z ============================= test session starts ==============================
2022-12-01T11:21:41.0283888Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:41.0284275Z cachedir: .pytest_cache
2022-12-01T11:21:41.0284833Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:41.0285254Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:21:41.0285583Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:41.0286145Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:41.0286560Z collecting ... collected 9 items
2022-12-01T11:21:41.0287869Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output
2022-12-01T11:21:41.0289051Z 
2022-12-01T11:21:41.0289291Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%]
2022-12-01T11:21:41.0289757Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%]
2022-12-01T11:21:41.0290480Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%]
2022-12-01T11:21:41.0291216Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%]
2022-12-01T11:21:41.0291734Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%]
2022-12-01T11:21:41.0292298Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%]
2022-12-01T11:21:41.0292797Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%]
2022-12-01T11:21:41.0293341Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%]
2022-12-01T11:21:41.0293851Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-12-01T11:21:41.0294089Z 
2022-12-01T11:21:41.0294252Z ============================== 9 passed in 1.37s ===============================
2022-12-01T11:21:41.0294684Z 
2022-12-01T11:21:41.0295020Z ##[endgroup]
2022-12-01T11:21:41.0295698Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_rhjrix_z)
2022-12-01T11:21:41.0296097Z 
2022-12-01T11:21:41.0296377Z Running distributed/pipeline/sync/test_dependency ... [2022-12-01 11:21:41.028109]
2022-12-01T11:21:41.0297012Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-12-01 11:21:41.028396]
2022-12-01T11:21:44.3901588Z 
2022-12-01T11:21:44.3902353Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency
2022-12-01T11:21:44.3903561Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_yvokzwns)
2022-12-01T11:21:44.3904092Z ============================= test session starts ==============================
2022-12-01T11:21:44.3904670Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:44.3905031Z cachedir: .pytest_cache
2022-12-01T11:21:44.3905622Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:44.3906096Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:21:44.3906395Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:44.3906951Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:44.3907345Z collecting ... collected 6 items
2022-12-01T11:21:44.3908247Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad
2022-12-01T11:21:44.3909019Z 
2022-12-01T11:21:44.3909241Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED      [ 16%]
2022-12-01T11:21:44.3909687Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2022-12-01T11:21:44.3910154Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2022-12-01T11:21:44.3910607Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2022-12-01T11:21:44.3911060Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2022-12-01T11:21:44.3911550Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2022-12-01T11:21:44.3911821Z 
2022-12-01T11:21:44.3911984Z ============================== 6 passed in 1.35s ===============================
2022-12-01T11:21:44.3912183Z 
2022-12-01T11:21:44.3912491Z ##[endgroup]
2022-12-01T11:21:44.3913405Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_yvokzwns)
2022-12-01T11:21:44.3913821Z 
2022-12-01T11:21:44.3914223Z Running distributed/pipeline/sync/test_phony ... [2022-12-01 11:21:44.390176]
2022-12-01T11:21:44.3914831Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-12-01 11:21:44.390474]
2022-12-01T11:21:46.3131792Z 
2022-12-01T11:21:46.3132303Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony
2022-12-01T11:21:46.3133325Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_jn4xblfc)
2022-12-01T11:21:46.3133834Z ============================= test session starts ==============================
2022-12-01T11:21:46.3135160Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:46.3135510Z cachedir: .pytest_cache
2022-12-01T11:21:46.3136087Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:46.3136554Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:21:46.3136877Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:46.3137876Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:46.3138180Z collecting ... collected 4 items
2022-12-01T11:21:46.3138944Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function
2022-12-01T11:21:46.3139403Z 
2022-12-01T11:21:46.3139618Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-12-01T11:21:46.3140065Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-12-01T11:21:46.3140503Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-12-01T11:21:46.3140941Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-12-01T11:21:46.3141206Z 
2022-12-01T11:21:46.3141364Z ============================== 4 passed in 0.04s ===============================
2022-12-01T11:21:46.3141564Z 
2022-12-01T11:21:46.3141882Z ##[endgroup]
2022-12-01T11:21:46.3142502Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_jn4xblfc)
2022-12-01T11:21:46.3142879Z 
2022-12-01T11:21:46.3143161Z Running distributed/pipeline/sync/test_stream ... [2022-12-01 11:21:46.313306]
2022-12-01T11:21:46.3143767Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-12-01 11:21:46.313632]
2022-12-01T11:21:50.7809294Z 
2022-12-01T11:21:50.7809780Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream
2022-12-01T11:21:50.7810809Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_fsc6u2x6)
2022-12-01T11:21:50.7811356Z ============================= test session starts ==============================
2022-12-01T11:21:50.7811967Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:21:50.7812308Z cachedir: .pytest_cache
2022-12-01T11:21:50.7812880Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:21:50.7813317Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:21:50.7813628Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:21:50.7814185Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:21:50.7815088Z collecting ... collected 19 items
2022-12-01T11:21:50.7817598Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view
2022-12-01T11:21:50.7820124Z 
2022-12-01T11:21:50.7820367Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2022-12-01T11:21:50.7820850Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%]
2022-12-01T11:21:50.7821331Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2022-12-01T11:21:50.7821854Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%]
2022-12-01T11:21:50.7822362Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2022-12-01T11:21:50.7822865Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%]
2022-12-01T11:21:50.7823331Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2022-12-01T11:21:50.7823807Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%]
2022-12-01T11:21:50.7824277Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2022-12-01T11:21:50.7824742Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%]
2022-12-01T11:21:50.7825195Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2022-12-01T11:21:50.7825665Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%]
2022-12-01T11:21:50.7826151Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2022-12-01T11:21:50.7826625Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%]
2022-12-01T11:21:50.7827111Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%]
2022-12-01T11:21:50.7827598Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%]
2022-12-01T11:21:50.7828089Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2022-12-01T11:21:50.7828559Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%]
2022-12-01T11:21:50.7829130Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%]
2022-12-01T11:21:50.7829428Z 
2022-12-01T11:21:50.7829589Z ============================== 19 passed in 2.50s ==============================
2022-12-01T11:21:50.7830001Z 
2022-12-01T11:21:50.7830267Z ##[endgroup]
2022-12-01T11:21:50.7830916Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_fsc6u2x6)
2022-12-01T11:21:50.7831293Z 
2022-12-01T11:21:50.7831562Z Running distributed/rpc/test_faulty_agent ... [2022-12-01 11:21:50.780969]
2022-12-01T11:21:50.7832265Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:21:50.781258]
2022-12-01T11:21:52.9028344Z 
2022-12-01T11:21:52.9029084Z Expand the folded group to see the log file of distributed/rpc/test_faulty_agent
2022-12-01T11:21:52.9030773Z ##[group]PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_t54u491t)
2022-12-01T11:21:52.9031383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnt1odvl
2022-12-01T11:21:52.9031951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnt1odvl/_remote_module_non_scriptable.py
2022-12-01T11:21:52.9032221Z 
2022-12-01T11:21:52.9032533Z ##[endgroup]
2022-12-01T11:21:52.9033291Z FINISHED PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_t54u491t)
2022-12-01T11:21:52.9033653Z 
2022-12-01T11:21:53.1857004Z 
2022-12-01T11:21:53.1857515Z real	59m25.119s
2022-12-01T11:21:53.1858116Z user	119m32.479s
2022-12-01T11:21:53.1858563Z sys	65m50.812s
2022-12-01T11:21:53.1859058Z + assert_git_not_dirty
2022-12-01T11:21:53.1859597Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-12-01T11:21:53.1860034Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2022-12-01T11:21:53.1860878Z ++ git status --porcelain
2022-12-01T11:21:54.7219323Z + git_status=
2022-12-01T11:21:54.7220150Z + [[ -n '' ]]
2022-12-01T11:21:54.7220826Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-12-01T11:21:54.7221137Z + [[ 2 == 1 ]]
2022-12-01T11:21:54.7221367Z + [[ 2 == 1 ]]
2022-12-01T11:21:54.7291328Z Prepare all required actions
2022-12-01T11:21:54.7291678Z Getting action download info
2022-12-01T11:21:54.9081985Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-12-01T11:21:54.9082270Z with:
2022-12-01T11:21:54.9082726Z   github-token: ***
2022-12-01T11:21:54.9082964Z env:
2022-12-01T11:21:54.9083186Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:54.9083453Z   GPU_FLAG: --gpus all
2022-12-01T11:21:54.9083701Z ##[endgroup]
2022-12-01T11:21:54.9137928Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-12-01T11:21:54.9138259Z with:
2022-12-01T11:21:54.9138489Z   shell: bash
2022-12-01T11:21:54.9138724Z   timeout_minutes: 10
2022-12-01T11:21:54.9138981Z   max_attempts: 5
2022-12-01T11:21:54.9139237Z   retry_wait_seconds: 30
2022-12-01T11:21:54.9139799Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-12-01T11:21:54.9140346Z   polling_interval_seconds: 1
2022-12-01T11:21:54.9140607Z   warning_on_retry: true
2022-12-01T11:21:54.9140875Z   continue_on_error: false
2022-12-01T11:21:54.9141125Z env:
2022-12-01T11:21:54.9141352Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:54.9141624Z   GPU_FLAG: --gpus all
2022-12-01T11:21:54.9142034Z   GITHUB_TOKEN: ***
2022-12-01T11:21:54.9142284Z ##[endgroup]
2022-12-01T11:21:54.9692612Z 
2022-12-01T11:21:54.9701608Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T11:21:54.9778115Z + python3 -m pip install requests==2.26.0
2022-12-01T11:21:55.2614885Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T11:21:55.3975260Z Collecting requests==2.26.0
2022-12-01T11:21:55.4138100Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-12-01T11:21:55.4971434Z Collecting certifi>=2017.4.17
2022-12-01T11:21:55.5013100Z   Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
2022-12-01T11:21:55.6093640Z Collecting urllib3<1.27,>=1.21.1
2022-12-01T11:21:55.6134349Z   Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
2022-12-01T11:21:55.6898334Z Collecting idna<4,>=2.5; python_version >= "3"
2022-12-01T11:21:55.6936003Z   Downloading idna-3.4-py3-none-any.whl (61 kB)
2022-12-01T11:21:55.8624709Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-12-01T11:21:55.8662720Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-12-01T11:21:55.9575465Z Installing collected packages: certifi, urllib3, idna, charset-normalizer, requests
2022-12-01T11:21:56.1609168Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-12-01T11:21:56.1609863Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T11:21:56.2087099Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.13
2022-12-01T11:21:56.2594940Z ++ python3 .github/scripts/get_workflow_job_id.py 3591403534 i-0513dad604108ca34
2022-12-01T11:21:57.9467982Z + GHA_WORKFLOW_JOB_ID=9818608475
2022-12-01T11:21:57.9469039Z + echo '::set-output name=job-id::9818608475'
2022-12-01T11:21:57.9471432Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T11:21:57.9758487Z Command completed after 1 attempt(s).
2022-12-01T11:21:57.9758704Z 
2022-12-01T11:21:57.9761049Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T11:21:57.9904548Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-12-01T11:21:57.9904906Z kill "$MONITOR_SCRIPT_PID"
2022-12-01T11:21:57.9918245Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:21:57.9918544Z env:
2022-12-01T11:21:57.9918786Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:57.9919035Z   GPU_FLAG: --gpus all
2022-12-01T11:21:57.9919298Z   MONITOR_SCRIPT_PID: 87857
2022-12-01T11:21:57.9919555Z ##[endgroup]
2022-12-01T11:21:58.0012803Z Prepare all required actions
2022-12-01T11:21:58.0013172Z Getting action download info
2022-12-01T11:21:58.2164966Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-12-01T11:21:58.3600072Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-12-01T11:21:58.3600389Z with:
2022-12-01T11:21:58.3600742Z   file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475
2022-12-01T11:21:58.3601066Z env:
2022-12-01T11:21:58.3601302Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:58.3601569Z   GPU_FLAG: --gpus all
2022-12-01T11:21:58.3601795Z ##[endgroup]
2022-12-01T11:21:58.3633705Z ##[group]Run # Remove any previous test jsons if they exist
2022-12-01T11:21:58.3634080Z # Remove any previous test jsons if they exist
2022-12-01T11:21:58.3634391Z rm -f test-jsons-*.zip
2022-12-01T11:21:58.3634720Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-12-01T11:21:58.3646824Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:21:58.3647122Z env:
2022-12-01T11:21:58.3647342Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:58.3647609Z   GPU_FLAG: --gpus all
2022-12-01T11:21:58.3647980Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475
2022-12-01T11:21:58.3648318Z ##[endgroup]
2022-12-01T11:21:58.3791454Z   adding: test/allowlist_for_publicAPI.json (deflated 80%)
2022-12-01T11:21:58.3825896Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-12-01T11:21:58.3833155Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-12-01T11:21:58.3834533Z   adding: test/.pytorch-slow-tests.json (deflated 76%)
2022-12-01T11:21:58.3845253Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-12-01T11:21:58.3871362Z ##[group]Run # Remove any previous test reports if they exist
2022-12-01T11:21:58.3871773Z # Remove any previous test reports if they exist
2022-12-01T11:21:58.3872100Z rm -f test-reports-*.zip
2022-12-01T11:21:58.3872439Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-12-01T11:21:58.3884133Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:21:58.3884428Z env:
2022-12-01T11:21:58.3884663Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:58.3884906Z   GPU_FLAG: --gpus all
2022-12-01T11:21:58.3885277Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475
2022-12-01T11:21:58.3885626Z ##[endgroup]
2022-12-01T11:21:58.3998912Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102236.xml (deflated 38%)
2022-12-01T11:21:58.3999606Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102242.xml (deflated 38%)
2022-12-01T11:21:58.4000274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102249.xml (deflated 38%)
2022-12-01T11:21:58.4000959Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102255.xml (deflated 38%)
2022-12-01T11:21:58.4001627Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102301.xml (deflated 38%)
2022-12-01T11:21:58.4002298Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102309.xml (deflated 38%)
2022-12-01T11:21:58.4002979Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102316.xml (deflated 39%)
2022-12-01T11:21:58.4003627Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102323.xml (deflated 38%)
2022-12-01T11:21:58.4004317Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102329.xml (deflated 38%)
2022-12-01T11:21:58.4004957Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102335.xml (deflated 38%)
2022-12-01T11:21:58.4005778Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20221201102341.xml (deflated 37%)
2022-12-01T11:21:58.4006467Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102347.xml (deflated 38%)
2022-12-01T11:21:58.4007173Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102353.xml (deflated 38%)
2022-12-01T11:21:58.4007877Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102401.xml (deflated 38%)
2022-12-01T11:21:58.4008569Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102407.xml (deflated 38%)
2022-12-01T11:21:58.4009238Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102414.xml (deflated 38%)
2022-12-01T11:21:58.4009925Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102420.xml (deflated 38%)
2022-12-01T11:21:58.4010626Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102428.xml (deflated 39%)
2022-12-01T11:21:58.4011324Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102434.xml (deflated 38%)
2022-12-01T11:21:58.4011971Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102442.xml (deflated 38%)
2022-12-01T11:21:58.4012767Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102448.xml (deflated 38%)
2022-12-01T11:21:58.4013455Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20221201102454.xml (deflated 38%)
2022-12-01T11:21:58.4014213Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102501.xml (deflated 45%)
2022-12-01T11:21:58.4015273Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102510.xml (deflated 45%)
2022-12-01T11:21:58.4016095Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102518.xml (deflated 43%)
2022-12-01T11:21:58.4016889Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102526.xml (deflated 43%)
2022-12-01T11:21:58.4017710Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102534.xml (deflated 45%)
2022-12-01T11:21:58.4018508Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102542.xml (deflated 46%)
2022-12-01T11:21:58.4019293Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102550.xml (deflated 47%)
2022-12-01T11:21:58.4020089Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102559.xml (deflated 47%)
2022-12-01T11:21:58.4020901Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102607.xml (deflated 44%)
2022-12-01T11:21:58.4021675Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102615.xml (deflated 46%)
2022-12-01T11:21:58.4022453Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102623.xml (deflated 46%)
2022-12-01T11:21:58.4023235Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102631.xml (deflated 44%)
2022-12-01T11:21:58.4024032Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102640.xml (deflated 44%)
2022-12-01T11:21:58.4024866Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102648.xml (deflated 43%)
2022-12-01T11:21:58.4025715Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102654.xml (deflated 44%)
2022-12-01T11:21:58.4026523Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102702.xml (deflated 45%)
2022-12-01T11:21:58.4027343Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102708.xml (deflated 44%)
2022-12-01T11:21:58.4028103Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102714.xml (deflated 45%)
2022-12-01T11:21:58.4028870Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102720.xml (deflated 45%)
2022-12-01T11:21:58.4029688Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102726.xml (deflated 50%)
2022-12-01T11:21:58.4030460Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102734.xml (deflated 42%)
2022-12-01T11:21:58.4031249Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102741.xml (deflated 41%)
2022-12-01T11:21:58.4032056Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102749.xml (deflated 41%)
2022-12-01T11:21:58.4032902Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102756.xml (deflated 41%)
2022-12-01T11:21:58.4033688Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102805.xml (deflated 42%)
2022-12-01T11:21:58.4034507Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102813.xml (deflated 42%)
2022-12-01T11:21:58.4035265Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102819.xml (deflated 42%)
2022-12-01T11:21:58.4036036Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102825.xml (deflated 41%)
2022-12-01T11:21:58.4036842Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102831.xml (deflated 41%)
2022-12-01T11:21:58.4037615Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102837.xml (deflated 44%)
2022-12-01T11:21:58.4038430Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102843.xml (deflated 45%)
2022-12-01T11:21:58.4039230Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102849.xml (deflated 41%)
2022-12-01T11:21:58.4039995Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102855.xml (deflated 41%)
2022-12-01T11:21:58.4040780Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102903.xml (deflated 42%)
2022-12-01T11:21:58.4041565Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102909.xml (deflated 42%)
2022-12-01T11:21:58.4042339Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102915.xml (deflated 42%)
2022-12-01T11:21:58.4043133Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20221201102925.xml (deflated 41%)
2022-12-01T11:21:58.4044028Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20221201102933.xml (deflated 43%)
2022-12-01T11:21:58.4044953Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102939.xml (deflated 39%)
2022-12-01T11:21:58.4045726Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102945.xml (deflated 40%)
2022-12-01T11:21:58.4046464Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102953.xml (deflated 39%)
2022-12-01T11:21:58.4047220Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201102959.xml (deflated 40%)
2022-12-01T11:21:58.4047980Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103005.xml (deflated 39%)
2022-12-01T11:21:58.4048721Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103012.xml (deflated 40%)
2022-12-01T11:21:58.4049456Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103018.xml (deflated 40%)
2022-12-01T11:21:58.4050212Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103025.xml (deflated 39%)
2022-12-01T11:21:58.4050960Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103034.xml (deflated 39%)
2022-12-01T11:21:58.4051781Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103040.xml (deflated 40%)
2022-12-01T11:21:58.4052510Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103049.xml (deflated 39%)
2022-12-01T11:21:58.4053267Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103057.xml (deflated 40%)
2022-12-01T11:21:58.4054001Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103103.xml (deflated 40%)
2022-12-01T11:21:58.4054927Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103109.xml (deflated 40%)
2022-12-01T11:21:58.4055653Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103115.xml (deflated 40%)
2022-12-01T11:21:58.4056398Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103122.xml (deflated 40%)
2022-12-01T11:21:58.4057147Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103128.xml (deflated 40%)
2022-12-01T11:21:58.4057888Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103136.xml (deflated 40%)
2022-12-01T11:21:58.4058614Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103142.xml (deflated 40%)
2022-12-01T11:21:58.4059365Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103149.xml (deflated 40%)
2022-12-01T11:21:58.4060136Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103157.xml (deflated 40%)
2022-12-01T11:21:58.4060882Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103203.xml (deflated 40%)
2022-12-01T11:21:58.4061603Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103209.xml (deflated 40%)
2022-12-01T11:21:58.4062339Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103217.xml (deflated 40%)
2022-12-01T11:21:58.4063078Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103224.xml (deflated 40%)
2022-12-01T11:21:58.4063819Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103230.xml (deflated 40%)
2022-12-01T11:21:58.4064629Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103238.xml (deflated 40%)
2022-12-01T11:21:58.4065397Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103244.xml (deflated 40%)
2022-12-01T11:21:58.4066138Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103250.xml (deflated 40%)
2022-12-01T11:21:58.4066914Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103258.xml (deflated 39%)
2022-12-01T11:21:58.4067655Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103305.xml (deflated 39%)
2022-12-01T11:21:58.4068436Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103311.xml (deflated 39%)
2022-12-01T11:21:58.4069145Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103318.xml (deflated 39%)
2022-12-01T11:21:58.4069920Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103328.xml (deflated 40%)
2022-12-01T11:21:58.4070602Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103334.xml (deflated 40%)
2022-12-01T11:21:58.4071448Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103340.xml (deflated 39%)
2022-12-01T11:21:58.4072176Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103348.xml (deflated 39%)
2022-12-01T11:21:58.4072918Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103354.xml (deflated 40%)
2022-12-01T11:21:58.4073642Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103401.xml (deflated 40%)
2022-12-01T11:21:58.4074381Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103410.xml (deflated 40%)
2022-12-01T11:21:58.4075117Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103416.xml (deflated 40%)
2022-12-01T11:21:58.4075863Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103422.xml (deflated 40%)
2022-12-01T11:21:58.4076577Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103429.xml (deflated 39%)
2022-12-01T11:21:58.4077318Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103437.xml (deflated 40%)
2022-12-01T11:21:58.4078059Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103443.xml (deflated 39%)
2022-12-01T11:21:58.4078802Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103450.xml (deflated 41%)
2022-12-01T11:21:58.4079522Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103452.xml (deflated 40%)
2022-12-01T11:21:58.4080291Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103458.xml (deflated 41%)
2022-12-01T11:21:58.4081012Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103500.xml (deflated 40%)
2022-12-01T11:21:58.4081753Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20221201103508.xml (deflated 40%)
2022-12-01T11:21:58.4082446Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103514.xml (deflated 38%)
2022-12-01T11:21:58.4083187Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103516.xml (deflated 39%)
2022-12-01T11:21:58.4083894Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103519.xml (deflated 39%)
2022-12-01T11:21:58.4084557Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103521.xml (deflated 39%)
2022-12-01T11:21:58.4085220Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103523.xml (deflated 38%)
2022-12-01T11:21:58.4085902Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20221201103525.xml (deflated 39%)
2022-12-01T11:21:58.4086605Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20221201103527.xml (deflated 39%)
2022-12-01T11:21:58.4087305Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20221201103531.xml (deflated 41%)
2022-12-01T11:21:58.4088102Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221201103536.xml (deflated 92%)
2022-12-01T11:21:58.4088997Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221201103536.xml (deflated 65%)
2022-12-01T11:21:58.4089886Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20221201104025.xml (deflated 93%)
2022-12-01T11:21:58.4090882Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20221201104025.xml (deflated 85%)
2022-12-01T11:21:58.4091780Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20221201104444.xml (deflated 91%)
2022-12-01T11:21:58.4092563Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221201104630.xml (deflated 81%)
2022-12-01T11:21:58.4093325Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221201104630.xml (deflated 89%)
2022-12-01T11:21:58.4094075Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201104802.xml (deflated 40%)
2022-12-01T11:21:58.4094982Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221201104805.xml (deflated 39%)
2022-12-01T11:21:58.4095680Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201104809.xml (deflated 39%)
2022-12-01T11:21:58.4096394Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221201104812.xml (deflated 39%)
2022-12-01T11:21:58.4097082Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201104815.xml (deflated 40%)
2022-12-01T11:21:58.4097822Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221201104819.xml (deflated 40%)
2022-12-01T11:21:58.4098531Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221201104822.xml (deflated 40%)
2022-12-01T11:21:58.4099258Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201104824.xml (deflated 40%)
2022-12-01T11:21:58.4099987Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221201104827.xml (deflated 39%)
2022-12-01T11:21:58.4100702Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221201104830.xml (deflated 39%)
2022-12-01T11:21:58.4101429Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221201104834.xml (deflated 39%)
2022-12-01T11:21:58.4102114Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201104837.xml (deflated 39%)
2022-12-01T11:21:58.4102908Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221201104840.xml (deflated 40%)
2022-12-01T11:21:58.4103638Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104844.xml (deflated 39%)
2022-12-01T11:21:58.4104337Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104847.xml (deflated 39%)
2022-12-01T11:21:58.4105084Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104850.xml (deflated 39%)
2022-12-01T11:21:58.4105796Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221201104854.xml (deflated 40%)
2022-12-01T11:21:58.4106455Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201104907.xml (deflated 38%)
2022-12-01T11:21:58.4107136Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221201104911.xml (deflated 38%)
2022-12-01T11:21:58.4107831Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104914.xml (deflated 38%)
2022-12-01T11:21:58.4108522Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104917.xml (deflated 39%)
2022-12-01T11:21:58.4109210Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104921.xml (deflated 38%)
2022-12-01T11:21:58.4109991Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104924.xml (deflated 38%)
2022-12-01T11:21:58.4110664Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104927.xml (deflated 38%)
2022-12-01T11:21:58.4111337Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104931.xml (deflated 39%)
2022-12-01T11:21:58.4112047Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104934.xml (deflated 38%)
2022-12-01T11:21:58.4112675Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221201104940.xml (deflated 39%)
2022-12-01T11:21:58.4113397Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20221201104943.xml (deflated 91%)
2022-12-01T11:21:58.4114212Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardGradScaler-20221201105020.xml (deflated 64%)
2022-12-01T11:21:58.4115099Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_sharded_grad_scaler/TEST-TestShardedGradScalerParityWithDDP-20221201105020.xml (deflated 84%)
2022-12-01T11:21:58.4115869Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20221201105058.xml (deflated 38%)
2022-12-01T11:21:58.4116640Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105104.xml (deflated 42%)
2022-12-01T11:21:58.4117445Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105108.xml (deflated 40%)
2022-12-01T11:21:58.4118247Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105111.xml (deflated 41%)
2022-12-01T11:21:58.4119057Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20221201105115.xml (deflated 42%)
2022-12-01T11:21:58.4119865Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105119.xml (deflated 42%)
2022-12-01T11:21:58.4120715Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105125.xml (deflated 42%)
2022-12-01T11:21:58.4121566Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105134.xml (deflated 41%)
2022-12-01T11:21:58.4122453Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20221201105140.xml (deflated 41%)
2022-12-01T11:21:58.4123239Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20221201105149.xml (deflated 39%)
2022-12-01T11:21:58.4123999Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221201105152.xml (deflated 86%)
2022-12-01T11:21:58.4124826Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221201105223.xml (deflated 80%)
2022-12-01T11:21:58.4125576Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221201105244.xml (deflated 55%)
2022-12-01T11:21:58.4126392Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221201105301.xml (deflated 43%)
2022-12-01T11:21:58.4127269Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221201105301.xml (deflated 43%)
2022-12-01T11:21:58.4128100Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20221201105325.xml (deflated 57%)
2022-12-01T11:21:58.4128898Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221201105336.xml (deflated 66%)
2022-12-01T11:21:58.4129810Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221201105336.xml (deflated 78%)
2022-12-01T11:21:58.4130571Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20221201105345.xml (deflated 55%)
2022-12-01T11:21:58.4131351Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221201105352.xml (deflated 47%)
2022-12-01T11:21:58.4132137Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221201105359.xml (deflated 40%)
2022-12-01T11:21:58.4132854Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20221201105406.xml (deflated 69%)
2022-12-01T11:21:58.4133676Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20221201105410.xml (deflated 68%)
2022-12-01T11:21:58.4134880Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20221201105410.xml (deflated 42%)
2022-12-01T11:21:58.4135948Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201105410.xml (deflated 45%)
2022-12-01T11:21:58.4136912Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_param_exec_order_wrap/TEST-TestFSDPExecOrder-20221201105423.xml (deflated 76%)
2022-12-01T11:21:58.4137777Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221201105443.xml (deflated 41%)
2022-12-01T11:21:58.4138676Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201105449.xml (deflated 41%)
2022-12-01T11:21:58.4139599Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201105458.xml (deflated 41%)
2022-12-01T11:21:58.4140480Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221201105506.xml (deflated 41%)
2022-12-01T11:21:58.4141371Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105514.xml (deflated 41%)
2022-12-01T11:21:58.4142337Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105522.xml (deflated 41%)
2022-12-01T11:21:58.4143260Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105530.xml (deflated 41%)
2022-12-01T11:21:58.4144138Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221201105536.xml (deflated 41%)
2022-12-01T11:21:58.4145008Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221201105544.xml (deflated 41%)
2022-12-01T11:21:58.4145853Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105554.xml (deflated 40%)
2022-12-01T11:21:58.4146719Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105600.xml (deflated 40%)
2022-12-01T11:21:58.4147587Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105606.xml (deflated 40%)
2022-12-01T11:21:58.4148492Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105612.xml (deflated 40%)
2022-12-01T11:21:58.4149403Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105618.xml (deflated 40%)
2022-12-01T11:21:58.4150262Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105624.xml (deflated 40%)
2022-12-01T11:21:58.4151124Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105630.xml (deflated 40%)
2022-12-01T11:21:58.4151992Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221201105636.xml (deflated 40%)
2022-12-01T11:21:58.4152904Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105642.xml (deflated 42%)
2022-12-01T11:21:58.4153848Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105656.xml (deflated 42%)
2022-12-01T11:21:58.4154815Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105709.xml (deflated 42%)
2022-12-01T11:21:58.4155771Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105722.xml (deflated 43%)
2022-12-01T11:21:58.4156759Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105734.xml (deflated 43%)
2022-12-01T11:21:58.4157699Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105745.xml (deflated 43%)
2022-12-01T11:21:58.4158643Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105757.xml (deflated 43%)
2022-12-01T11:21:58.4159579Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105809.xml (deflated 43%)
2022-12-01T11:21:58.4160527Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105820.xml (deflated 43%)
2022-12-01T11:21:58.4161456Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105832.xml (deflated 43%)
2022-12-01T11:21:58.4162462Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105838.xml (deflated 43%)
2022-12-01T11:21:58.4163437Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105844.xml (deflated 43%)
2022-12-01T11:21:58.4164398Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105850.xml (deflated 43%)
2022-12-01T11:21:58.4165345Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105856.xml (deflated 43%)
2022-12-01T11:21:58.4166296Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105904.xml (deflated 43%)
2022-12-01T11:21:58.4167221Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105912.xml (deflated 42%)
2022-12-01T11:21:58.4168225Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105925.xml (deflated 43%)
2022-12-01T11:21:58.4169183Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105928.xml (deflated 42%)
2022-12-01T11:21:58.4170176Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201105950.xml (deflated 43%)
2022-12-01T11:21:58.4171120Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110005.xml (deflated 43%)
2022-12-01T11:21:58.4172069Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110019.xml (deflated 43%)
2022-12-01T11:21:58.4173025Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110025.xml (deflated 42%)
2022-12-01T11:21:58.4173983Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110035.xml (deflated 42%)
2022-12-01T11:21:58.4175085Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110044.xml (deflated 43%)
2022-12-01T11:21:58.4176035Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110054.xml (deflated 43%)
2022-12-01T11:21:58.4176981Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110058.xml (deflated 43%)
2022-12-01T11:21:58.4177932Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110110.xml (deflated 42%)
2022-12-01T11:21:58.4178871Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110122.xml (deflated 43%)
2022-12-01T11:21:58.4179821Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110134.xml (deflated 43%)
2022-12-01T11:21:58.4180765Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110145.xml (deflated 43%)
2022-12-01T11:21:58.4181717Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110157.xml (deflated 42%)
2022-12-01T11:21:58.4182743Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110209.xml (deflated 42%)
2022-12-01T11:21:58.4183689Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110221.xml (deflated 43%)
2022-12-01T11:21:58.4184639Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110233.xml (deflated 42%)
2022-12-01T11:21:58.4185594Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110245.xml (deflated 42%)
2022-12-01T11:21:58.4186542Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110257.xml (deflated 42%)
2022-12-01T11:21:58.4187465Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110309.xml (deflated 42%)
2022-12-01T11:21:58.4188409Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110321.xml (deflated 42%)
2022-12-01T11:21:58.4189355Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110333.xml (deflated 42%)
2022-12-01T11:21:58.4190792Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110345.xml (deflated 42%)
2022-12-01T11:21:58.4192187Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110357.xml (deflated 42%)
2022-12-01T11:21:58.4193122Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110409.xml (deflated 42%)
2022-12-01T11:21:58.4194076Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110418.xml (deflated 43%)
2022-12-01T11:21:58.4195027Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110430.xml (deflated 42%)
2022-12-01T11:21:58.4195978Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110440.xml (deflated 42%)
2022-12-01T11:21:58.4196902Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110449.xml (deflated 42%)
2022-12-01T11:21:58.4197891Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110501.xml (deflated 43%)
2022-12-01T11:21:58.4198844Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110513.xml (deflated 43%)
2022-12-01T11:21:58.4199788Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110519.xml (deflated 43%)
2022-12-01T11:21:58.4200733Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110526.xml (deflated 43%)
2022-12-01T11:21:58.4201676Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110532.xml (deflated 42%)
2022-12-01T11:21:58.4202606Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110538.xml (deflated 42%)
2022-12-01T11:21:58.4203558Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110546.xml (deflated 42%)
2022-12-01T11:21:58.4204591Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110555.xml (deflated 42%)
2022-12-01T11:21:58.4205546Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110603.xml (deflated 42%)
2022-12-01T11:21:58.4206475Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110612.xml (deflated 42%)
2022-12-01T11:21:58.4207417Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110620.xml (deflated 42%)
2022-12-01T11:21:58.4208363Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110628.xml (deflated 42%)
2022-12-01T11:21:58.4209313Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110637.xml (deflated 42%)
2022-12-01T11:21:58.4210255Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110649.xml (deflated 42%)
2022-12-01T11:21:58.4211258Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110701.xml (deflated 43%)
2022-12-01T11:21:58.4212206Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110707.xml (deflated 43%)
2022-12-01T11:21:58.4213156Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110719.xml (deflated 43%)
2022-12-01T11:21:58.4214107Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110725.xml (deflated 43%)
2022-12-01T11:21:58.4215305Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110731.xml (deflated 42%)
2022-12-01T11:21:58.4216251Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110737.xml (deflated 43%)
2022-12-01T11:21:58.4217205Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110747.xml (deflated 42%)
2022-12-01T11:21:58.4218151Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110753.xml (deflated 42%)
2022-12-01T11:21:58.4219076Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110759.xml (deflated 42%)
2022-12-01T11:21:58.4220026Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110812.xml (deflated 42%)
2022-12-01T11:21:58.4220973Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110826.xml (deflated 42%)
2022-12-01T11:21:58.4221917Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110840.xml (deflated 42%)
2022-12-01T11:21:58.4222860Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110852.xml (deflated 42%)
2022-12-01T11:21:58.4223785Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110911.xml (deflated 43%)
2022-12-01T11:21:58.4224823Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110934.xml (deflated 42%)
2022-12-01T11:21:58.4225789Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110937.xml (deflated 42%)
2022-12-01T11:21:58.4226740Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201110959.xml (deflated 42%)
2022-12-01T11:21:58.4227664Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111019.xml (deflated 42%)
2022-12-01T11:21:58.4228605Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111037.xml (deflated 42%)
2022-12-01T11:21:58.4229554Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111055.xml (deflated 43%)
2022-12-01T11:21:58.4230501Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111114.xml (deflated 42%)
2022-12-01T11:21:58.4231447Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111132.xml (deflated 42%)
2022-12-01T11:21:58.4232465Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111152.xml (deflated 43%)
2022-12-01T11:21:58.4233411Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111213.xml (deflated 43%)
2022-12-01T11:21:58.4234358Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111233.xml (deflated 42%)
2022-12-01T11:21:58.4235306Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111255.xml (deflated 43%)
2022-12-01T11:21:58.4236235Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221201111304.xml (deflated 44%)
2022-12-01T11:21:58.4237205Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201111315.xml (deflated 44%)
2022-12-01T11:21:58.4238187Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201111322.xml (deflated 44%)
2022-12-01T11:21:58.4239177Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221201111328.xml (deflated 44%)
2022-12-01T11:21:58.4240172Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerDistributed-20221201111334.xml (deflated 90%)
2022-12-01T11:21:58.4241143Z   adding: test/test-reports/python-unittest/distributed.optim.test_zero_redundancy_optimizer/TEST-TestZeroRedundancyOptimizerSingleRank-20221201111334.xml (deflated 73%)
2022-12-01T11:21:58.4242054Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111732.xml (deflated 42%)
2022-12-01T11:21:58.4242899Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111740.xml (deflated 42%)
2022-12-01T11:21:58.4243743Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111748.xml (deflated 42%)
2022-12-01T11:21:58.4244621Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111757.xml (deflated 43%)
2022-12-01T11:21:58.4245470Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111805.xml (deflated 42%)
2022-12-01T11:21:58.4246311Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111813.xml (deflated 43%)
2022-12-01T11:21:58.4247161Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111821.xml (deflated 42%)
2022-12-01T11:21:58.4247980Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111829.xml (deflated 42%)
2022-12-01T11:21:58.4248811Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221201111837.xml (deflated 42%)
2022-12-01T11:21:58.4249582Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221201111844.xml (deflated 79%)
2022-12-01T11:21:58.4250375Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221201111928.xml (deflated 55%)
2022-12-01T11:21:58.4251225Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221201111928.xml (deflated 81%)
2022-12-01T11:21:58.4252206Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221201111928.xml (deflated 80%)
2022-12-01T11:21:58.4253014Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221201111928.xml (deflated 64%)
2022-12-01T11:21:58.4253813Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221201111953.xml (deflated 75%)
2022-12-01T11:21:58.4254880Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20221201112006.xml (deflated 73%)
2022-12-01T11:21:58.4255797Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20221201112017.xml (deflated 74%)
2022-12-01T11:21:58.4256688Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20221201112028.xml (deflated 69%)
2022-12-01T11:21:58.4257545Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221201112037.xml (deflated 60%)
2022-12-01T11:21:58.4258394Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221201112044.xml (deflated 60%)
2022-12-01T11:21:58.4259155Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221201112053.xml (deflated 83%)
2022-12-01T11:21:58.4259958Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20221201112053.xml (deflated 90%)
2022-12-01T11:21:58.4260805Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20221201112100.xml (deflated 53%)
2022-12-01T11:21:58.4261706Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221201112105.xml (deflated 44%)
2022-12-01T11:21:58.4262529Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221201112110.xml (deflated 55%)
2022-12-01T11:21:58.4263306Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221201112114.xml (deflated 63%)
2022-12-01T11:21:58.4264035Z   adding: test/test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221201112119.xml (deflated 83%)
2022-12-01T11:21:58.4264919Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111615.xml (deflated 44%)
2022-12-01T11:21:58.4265797Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111616.xml (deflated 44%)
2022-12-01T11:21:58.4266684Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111618.xml (deflated 42%)
2022-12-01T11:21:58.4267562Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111626.xml (deflated 42%)
2022-12-01T11:21:58.4268477Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111634.xml (deflated 42%)
2022-12-01T11:21:58.4269350Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111642.xml (deflated 42%)
2022-12-01T11:21:58.4270205Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111652.xml (deflated 44%)
2022-12-01T11:21:58.4271069Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111653.xml (deflated 44%)
2022-12-01T11:21:58.4272025Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111655.xml (deflated 42%)
2022-12-01T11:21:58.4272896Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111704.xml (deflated 42%)
2022-12-01T11:21:58.4273749Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111712.xml (deflated 42%)
2022-12-01T11:21:58.4274621Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111719.xml (deflated 42%)
2022-12-01T11:21:58.4275489Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111615.xml (deflated 41%)
2022-12-01T11:21:58.4276364Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111620.xml (deflated 42%)
2022-12-01T11:21:58.4277209Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111625.xml (deflated 44%)
2022-12-01T11:21:58.4278075Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111627.xml (deflated 44%)
2022-12-01T11:21:58.4278942Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111629.xml (deflated 45%)
2022-12-01T11:21:58.4279806Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111631.xml (deflated 45%)
2022-12-01T11:21:58.4280650Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111652.xml (deflated 41%)
2022-12-01T11:21:58.4281514Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111657.xml (deflated 42%)
2022-12-01T11:21:58.4282373Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111702.xml (deflated 44%)
2022-12-01T11:21:58.4283236Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111704.xml (deflated 44%)
2022-12-01T11:21:58.4284162Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111706.xml (deflated 45%)
2022-12-01T11:21:58.4285030Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221201111708.xml (deflated 45%)
2022-12-01T11:21:58.4307538Z ##[group]Run # Remove any previous test reports if they exist
2022-12-01T11:21:58.4307937Z # Remove any previous test reports if they exist
2022-12-01T11:21:58.4308254Z rm -f usage-log-*.zip
2022-12-01T11:21:58.4308628Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-12-01T11:21:58.4309020Z # so check to see if the file exists first
2022-12-01T11:21:58.4309309Z if [ -f 'usage_log.txt' ]; then
2022-12-01T11:21:58.4309642Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-12-01T11:21:58.4309939Z fi
2022-12-01T11:21:58.4321444Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:21:58.4321747Z env:
2022-12-01T11:21:58.4321989Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:58.4322239Z   GPU_FLAG: --gpus all
2022-12-01T11:21:58.4322607Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475
2022-12-01T11:21:58.4322962Z ##[endgroup]
2022-12-01T11:21:58.4913141Z   adding: usage_log.txt (deflated 95%)
2022-12-01T11:21:58.4979148Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T11:21:58.4979444Z with:
2022-12-01T11:21:58.4979708Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T11:21:58.4980010Z   retention-days: 14
2022-12-01T11:21:58.4980280Z   if-no-files-found: warn
2022-12-01T11:21:58.4980537Z   path: test-jsons-*.zip
2022-12-01T11:21:58.4980790Z   name: artifact
2022-12-01T11:21:58.4981041Z   s3-bucket: gha-artifacts
2022-12-01T11:21:58.4981285Z   region: us-east-1
2022-12-01T11:21:58.4981514Z env:
2022-12-01T11:21:58.4981751Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:58.4981997Z   GPU_FLAG: --gpus all
2022-12-01T11:21:58.4982255Z ##[endgroup]
2022-12-01T11:21:58.9468721Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T11:21:58.9469322Z With the provided path, there will be 1 file uploaded
2022-12-01T11:21:58.9469708Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T11:21:58.9480556Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475.zip
2022-12-01T11:21:59.0686035Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475.zip
2022-12-01T11:21:59.0870016Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T11:21:59.0870311Z with:
2022-12-01T11:21:59.0870571Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T11:21:59.0870873Z   retention-days: 14
2022-12-01T11:21:59.0871144Z   if-no-files-found: error
2022-12-01T11:21:59.0871407Z   path: test-reports-*.zip
2022-12-01T11:21:59.0871669Z   name: artifact
2022-12-01T11:21:59.0871919Z   s3-bucket: gha-artifacts
2022-12-01T11:21:59.0872177Z   region: us-east-1
2022-12-01T11:21:59.0872432Z env:
2022-12-01T11:21:59.0872780Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:59.0873160Z   GPU_FLAG: --gpus all
2022-12-01T11:21:59.0873396Z ##[endgroup]
2022-12-01T11:21:59.5314837Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T11:21:59.5315853Z With the provided path, there will be 1 file uploaded
2022-12-01T11:21:59.5316291Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T11:21:59.5327142Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475.zip
2022-12-01T11:21:59.7526385Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475.zip
2022-12-01T11:21:59.7682252Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T11:21:59.7682549Z with:
2022-12-01T11:21:59.7682812Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T11:21:59.7683109Z   retention-days: 14
2022-12-01T11:21:59.7683376Z   if-no-files-found: ignore
2022-12-01T11:21:59.7683631Z   path: usage-log-*.zip
2022-12-01T11:21:59.7684013Z   name: artifact
2022-12-01T11:21:59.7684280Z   s3-bucket: gha-artifacts
2022-12-01T11:21:59.7684522Z   region: us-east-1
2022-12-01T11:21:59.7684752Z env:
2022-12-01T11:21:59.7684986Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:21:59.7685230Z   GPU_FLAG: --gpus all
2022-12-01T11:21:59.7685473Z ##[endgroup]
2022-12-01T11:22:00.2249536Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T11:22:00.2250492Z With the provided path, there will be 1 file uploaded
2022-12-01T11:22:00.2250871Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T11:22:00.2262428Z Starting upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475.zip
2022-12-01T11:22:00.4203940Z Finished upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_9818608475.zip
2022-12-01T11:22:00.4366685Z ##[group]Run set -x
2022-12-01T11:22:00.4366985Z set -x
2022-12-01T11:22:00.4367268Z python3 -m pip install -r requirements.txt
2022-12-01T11:22:00.4367607Z python3 -m pip install boto3==1.19.12
2022-12-01T11:22:00.4368068Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-12-01T11:22:00.4381678Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:22:00.4381964Z env:
2022-12-01T11:22:00.4382204Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:22:00.4382606Z   GPU_FLAG: --gpus all
2022-12-01T11:22:00.4382855Z   AWS_DEFAULT_REGION: us-east-1
2022-12-01T11:22:00.4383121Z   BRANCH: pull/89997
2022-12-01T11:22:00.4383378Z   TEST_CONFIG: distributed
2022-12-01T11:22:00.4383612Z   SHARD_NUMBER: 2
2022-12-01T11:22:00.4383927Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T11:22:00.4384242Z   PR_NUMBER: 89997
2022-12-01T11:22:00.4384482Z   PYTORCH_RETRY_TEST_CASES: 1
2022-12-01T11:22:00.4384803Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-12-01T11:22:00.4385127Z   SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T11:22:00.4385383Z   TAG: 
2022-12-01T11:22:00.4385612Z   WORKFLOW_ID: 3591403534
2022-12-01T11:22:00.4386040Z   GITHUB_TOKEN: ***
2022-12-01T11:22:00.4386285Z   GHA_WORKFLOW_JOB_ID: 9818608475
2022-12-01T11:22:00.4386546Z ##[endgroup]
2022-12-01T11:22:00.4417224Z + python3 -m pip install -r requirements.txt
2022-12-01T11:22:00.7321608Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T11:22:00.8123099Z Collecting astunparse
2022-12-01T11:22:00.8265066Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-12-01T11:22:00.8580766Z Collecting expecttest
2022-12-01T11:22:00.8618805Z   Downloading expecttest-0.1.4-py3-none-any.whl (6.5 kB)
2022-12-01T11:22:00.8999000Z Collecting future
2022-12-01T11:22:00.9037333Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-12-01T11:22:02.7905500Z Collecting hypothesis
2022-12-01T11:22:02.7955914Z   Downloading hypothesis-6.58.2-py3-none-any.whl (396 kB)
2022-12-01T11:22:03.6248015Z Collecting numpy
2022-12-01T11:22:03.6298809Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-12-01T11:22:03.9505779Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-12-01T11:22:04.0700473Z Collecting pyyaml
2022-12-01T11:22:04.0746755Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-12-01T11:22:04.0949349Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-12-01T11:22:04.1131425Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-12-01T11:22:04.1733730Z Collecting six
2022-12-01T11:22:04.1773643Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-12-01T11:22:04.2113302Z Collecting types-dataclasses
2022-12-01T11:22:04.2358235Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2022-12-01T11:22:04.2772308Z Collecting typing_extensions
2022-12-01T11:22:04.2980632Z   Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
2022-12-01T11:22:04.3528519Z Collecting sympy
2022-12-01T11:22:04.3583590Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2022-12-01T11:22:04.5874932Z Collecting wheel<1.0,>=0.23.0
2022-12-01T11:22:04.5911274Z   Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
2022-12-01T11:22:04.6357526Z Collecting sortedcontainers<3.0.0,>=2.1.0
2022-12-01T11:22:04.6395180Z   Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
2022-12-01T11:22:04.6758080Z Collecting exceptiongroup>=1.0.0; python_version < "3.11"
2022-12-01T11:22:04.6801429Z   Downloading exceptiongroup-1.0.4-py3-none-any.whl (14 kB)
2022-12-01T11:22:04.7291921Z Collecting attrs>=19.2.0
2022-12-01T11:22:04.7329781Z   Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
2022-12-01T11:22:04.7789587Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-12-01T11:22:04.7802506Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.13)
2022-12-01T11:22:04.8028109Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-12-01T11:22:04.8055178Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-12-01T11:22:04.8330345Z Collecting mpmath>=0.19
2022-12-01T11:22:04.8394623Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2022-12-01T11:22:04.8647756Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-12-01T11:22:05.0069803Z Installing collected packages: six, wheel, astunparse, expecttest, future, sortedcontainers, exceptiongroup, attrs, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy
2022-12-01T11:22:05.0471976Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-12-01T11:22:05.0472589Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T11:22:05.0767364Z     Running setup.py install for future: started
2022-12-01T11:22:05.7322394Z     Running setup.py install for future: finished with status 'done'
2022-12-01T11:22:06.0370099Z   WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-12-01T11:22:06.0370758Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T11:22:08.0208754Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-12-01T11:22:08.0209457Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T11:22:16.9965859Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-12-01T11:22:16.9966518Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T11:22:17.0266845Z Successfully installed astunparse-1.6.3 attrs-22.1.0 exceptiongroup-1.0.4 expecttest-0.1.4 future-0.18.2 hypothesis-6.58.2 mpmath-1.2.1 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.4.0 wheel-0.38.4
2022-12-01T11:22:17.1017100Z + python3 -m pip install boto3==1.19.12
2022-12-01T11:22:17.3913421Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T11:22:18.3857390Z Collecting boto3==1.19.12
2022-12-01T11:22:18.4128043Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-12-01T11:22:18.4683735Z Collecting jmespath<1.0.0,>=0.7.1
2022-12-01T11:22:18.4738554Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-12-01T11:22:18.5328308Z Collecting s3transfer<0.6.0,>=0.5.0
2022-12-01T11:22:18.5368084Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-12-01T11:22:19.7235112Z Collecting botocore<1.23.0,>=1.22.12
2022-12-01T11:22:19.7290687Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-12-01T11:22:19.9641268Z Collecting python-dateutil<3.0.0,>=2.1
2022-12-01T11:22:19.9777229Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-12-01T11:22:19.9956947Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.13)
2022-12-01T11:22:20.0183906Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-12-01T11:22:20.2014489Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2022-12-01T11:22:21.0919151Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-12-01T11:22:21.1481623Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-12-01T11:22:30.4144646Z [scribe] Scribe access token not provided, sending report via boto3...
2022-12-01T11:22:30.4145330Z 
2022-12-01T11:22:30.4148163Z ----- Historic stats comparison result ------
2022-12-01T11:22:30.4148406Z 
2022-12-01T11:22:30.4148931Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T11:22:30.4149307Z     commit: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T11:22:30.4149523Z 
2022-12-01T11:22:30.4149747Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-12-01T11:22:30.4151702Z 
2022-12-01T11:22:30.4153789Z     : (master)
2022-12-01T11:22:30.4154147Z     |
2022-12-01T11:22:30.4154483Z     | * c13d400bff (HEAD)            total time  2798.73s
2022-12-01T11:22:30.4154718Z     | |
2022-12-01T11:22:30.4154944Z     | : (59 commits)
2022-12-01T11:22:30.4155157Z     |/
2022-12-01T11:22:30.4156137Z     * 67eb2d5952 (base)  18 reports, total time  3503.90s ± 2079.90s
2022-12-01T11:22:30.4156599Z     * 1c5ca724f4          9 reports, total time  3529.71s ± 2080.27s
2022-12-01T11:22:30.4157074Z     * 9d6109c4b0          9 reports, total time  3530.22s ± 2110.41s
2022-12-01T11:22:30.4157500Z     * 736adc0808          9 reports, total time  3501.80s ± 2137.81s
2022-12-01T11:22:30.4157898Z     * a348975e00          9 reports, total time  3551.46s ± 2118.11s
2022-12-01T11:22:30.4158319Z     * db13049b88          9 reports, total time  3520.57s ± 2066.49s
2022-12-01T11:22:30.4158788Z     * d07b85393a          9 reports, total time  3540.86s ± 2074.85s
2022-12-01T11:22:30.4159213Z     * ac25c210e5          9 reports, total time  3549.18s ± 2105.70s
2022-12-01T11:22:30.4159636Z     * 2355b6256b          9 reports, total time  3596.87s ± 2152.28s
2022-12-01T11:22:30.4161901Z     * 4f95f7ae9b          9 reports, total time  3534.00s ± 2151.51s
2022-12-01T11:22:30.4162161Z     |
2022-12-01T11:22:30.4162359Z     :
2022-12-01T11:22:30.4162494Z 
2022-12-01T11:22:30.4162671Z Removed  (across  908 suites)     0 tests, totaling     0.00s
2022-12-01T11:22:30.4163065Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-12-01T11:22:30.4163447Z Added    (across   76 suites)   723 tests, totaling +2798.73s
2022-12-01T11:22:30.4721207Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-12-01T11:22:30.4721540Z with:
2022-12-01T11:22:30.4721753Z env:
2022-12-01T11:22:30.4721991Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:22:30.4722237Z   GPU_FLAG: --gpus all
2022-12-01T11:22:30.4722480Z ##[endgroup]
2022-12-01T11:22:30.4742573Z ##[group]Run set -eou pipefail
2022-12-01T11:22:30.4742880Z set -eou pipefail
2022-12-01T11:22:30.4743125Z 
2022-12-01T11:22:30.4743440Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-12-01T11:22:30.4743759Z for _ in $(seq 1440); do
2022-12-01T11:22:30.4744060Z     # Break if no ssh session exists anymore
2022-12-01T11:22:30.4744355Z     if [ "$(who)" = "" ]; then
2022-12-01T11:22:30.4744585Z       break
2022-12-01T11:22:30.4744813Z     fi
2022-12-01T11:22:30.4745054Z     echo "."
2022-12-01T11:22:30.4745275Z     sleep 5
2022-12-01T11:22:30.4745540Z done
2022-12-01T11:22:30.4758768Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:22:30.4759047Z env:
2022-12-01T11:22:30.4759284Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:22:30.4759552Z   GPU_FLAG: --gpus all
2022-12-01T11:22:30.4759778Z ##[endgroup]
2022-12-01T11:22:30.4790576Z Holding runner for 2 hours until all ssh sessions have logged out
2022-12-01T11:22:30.4864540Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-12-01T11:22:30.4864967Z # ignore expansion of "docker ps -q" since it could be empty
2022-12-01T11:22:30.4865305Z # shellcheck disable=SC2046
2022-12-01T11:22:30.4865613Z docker stop $(docker ps -q) || true
2022-12-01T11:22:30.4865906Z # Prune all of the docker images
2022-12-01T11:22:30.4866203Z docker system prune -af
2022-12-01T11:22:30.4878689Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:22:30.4878969Z env:
2022-12-01T11:22:30.4879213Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:22:30.4879485Z   GPU_FLAG: --gpus all
2022-12-01T11:22:30.4879716Z ##[endgroup]
2022-12-01T11:22:31.7741892Z 76d0c0aaee30
2022-12-01T11:22:32.5349107Z Deleted Containers:
2022-12-01T11:22:32.5349489Z 76d0c0aaee307e2ea54f34b360ef1b6ca5de6826777d6857a4e0a84ac40c4542
2022-12-01T11:22:32.5349748Z 
2022-12-01T11:22:37.5696768Z Deleted Images:
2022-12-01T11:22:37.5698302Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T11:22:37.5700029Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:217fd7de680e1dd5bca4b2b4054bd05a8d454df5e210ffbf1e5955e01cf1f340
2022-12-01T11:22:37.5700710Z deleted: sha256:f1c05e1c4a24b2f30e4b2dd175e18efaf97173a54d9813ad2e6088a740690342
2022-12-01T11:22:37.5701196Z deleted: sha256:1e7def21582cf03150a2e3ad04e2701388eaa422b3720fe7f648241a8988388e
2022-12-01T11:22:37.5701639Z deleted: sha256:3a7e72ba714442f54696c557b74dca88c618446caa68ff6506c2ce93cd7b2a85
2022-12-01T11:22:37.5702039Z deleted: sha256:acc47a47b0a4383dfbd2bd91be28c265a6486f388812591a0a9c185c62047c0a
2022-12-01T11:22:37.5702528Z deleted: sha256:60f22f4a0a4afdbef80bed81f20ae4c7a54233808a9544253a3c30a2776d2f71
2022-12-01T11:22:37.5702989Z deleted: sha256:5efa4324ff86638bcfa35ab9b1debaddf3d7513732c27ca8dd7e60e8358da6c9
2022-12-01T11:22:37.5703446Z deleted: sha256:a7f286562f5d9f9a1490bbc0a8125ba19aefca94af08516be2493eeb191f127f
2022-12-01T11:22:37.5703872Z deleted: sha256:e403df4fa1d82e23b8db68b6e2b5562fcd55535dd72214b9421e7814eee51ffd
2022-12-01T11:22:37.5704318Z deleted: sha256:f9471dd8edb7c2e1eb3aefda26ec6e67ee49347262f97f3111f176eeb2bf9465
2022-12-01T11:22:37.5704767Z deleted: sha256:d6df7f8a51a6e7483b61b1f330f3a4417b815fce5a7ad4c2a89510a54f5537da
2022-12-01T11:22:37.5705180Z deleted: sha256:b12c66528bc64f807b7a52116ecdf950025b66a403dd328a33db5511b63fc3f1
2022-12-01T11:22:37.5705805Z deleted: sha256:e8a84d179a24426205a63771d9aa894a6889147decc6d227312415e2842d0e38
2022-12-01T11:22:37.5706230Z deleted: sha256:b1e802e06103bb69d2a413954707d2f281a894d36a1f7fa3a65efa626a0e5e43
2022-12-01T11:22:37.5706661Z deleted: sha256:8efc9a5c3ed0496241d3ce1f76339b4d1270b5470ce1429effc703ca61a04c50
2022-12-01T11:22:37.5707057Z deleted: sha256:fef4f8c199945077358334d285e9dbe762a116427ec29d567139bee51863bef4
2022-12-01T11:22:37.5707478Z deleted: sha256:3f9f9d5fa745067d0ddf7f1206fcf51b87304adb8d32c87e130cbaae4acee275
2022-12-01T11:22:37.5707918Z deleted: sha256:75fb962b88e95b55a260c8cebb5da299635f32f3b104d08b08a2e1fba788437b
2022-12-01T11:22:37.5708339Z deleted: sha256:bd2c58efdd30f071840e8b3fd569126cf238bfdd5f0cd18345e36b60f93489df
2022-12-01T11:22:37.5708796Z deleted: sha256:d72aae1f9d4b7e1acc7b1954fed6c4886c49cec893a48dac5337fee3c9bf4c08
2022-12-01T11:22:37.5709240Z deleted: sha256:2b0461664863c4ef4ad02da7c9ec92a1d6d08bc8a9b76b5a982b7a98549980d3
2022-12-01T11:22:37.5709666Z deleted: sha256:086197a6cf735060ac5a6e595333f0bc14643bfddd3c338439c22002c1bec257
2022-12-01T11:22:37.5710101Z deleted: sha256:64254e4d362f949e9732531c579ca041aa47d4f70f7c242fda19048d82df0a1d
2022-12-01T11:22:37.5710827Z deleted: sha256:25374ed655bf3ce154d4ee14bd8c4f3220431d67b0d18acf3efdd666bbc01285
2022-12-01T11:22:37.5711461Z deleted: sha256:70a8b1a97ab5caea7ba2d2ecea43b58781672c588ee6375271d2b5f04d31594c
2022-12-01T11:22:37.5711895Z deleted: sha256:758af05d13f6edc84594db4db15f7b8066c5da10f6d83717a1d8b6ae5a2caccf
2022-12-01T11:22:37.5712534Z deleted: sha256:b37c30a9368604ae9d1873491a752ae71aba89de73dec5588638c1d2e3917901
2022-12-01T11:22:37.5713228Z deleted: sha256:790f109bf6ce127a4f3e25ea1675387993204350100fca6cecb56f8b863ce2e5
2022-12-01T11:22:37.5713933Z deleted: sha256:df13ec8c62d036e4b9fe0fd768e7fae7c5f76430b4aae4f3f356d701621b86d8
2022-12-01T11:22:37.5714677Z deleted: sha256:ef027ba5732bd4bc180cc8f8d769cd28630e92609a88be8f472e4c8ee8a3bb6d
2022-12-01T11:22:37.5715561Z deleted: sha256:5751424dd4ee3deaaf3e9f8b8172683ff7f30d90dc2415c3833042333f52b619
2022-12-01T11:22:37.5716278Z deleted: sha256:ac0364f6b4441a779e12c1527f256b052fff92b14aca383c87e62a8d6e4d1470
2022-12-01T11:22:37.5716924Z deleted: sha256:1f8161738ca37b8a92ed2577a6d9705167fb325ad8f8512439b47251013af048
2022-12-01T11:22:37.5717618Z deleted: sha256:2f8770a5227a4d66f9bdea07ca3e6e7cda5b06fbc362b354ae29f7542b3e7a19
2022-12-01T11:22:37.5718343Z deleted: sha256:652470cd61eae170df59d88a3b038cd2ab199526f4ba54badf85dd5628d7cb37
2022-12-01T11:22:37.5719073Z deleted: sha256:155eec0e8b217d2bafa06b3f85e987c3164bbad427de092045ddbff33f091a1f
2022-12-01T11:22:37.5719785Z deleted: sha256:8c60c1d76e4c0216ef0cc9248e31a656ab0a29bc95f589f4b23a0634523058c7
2022-12-01T11:22:37.5720453Z deleted: sha256:2811ebe06c45b7997659f3d50562406d001f3e979f398f9b4844bac84b3c419e
2022-12-01T11:22:37.5721144Z deleted: sha256:6a690d8e14cafcab7ab578a42baf2ca075033cf306f03e13ae70150b8be2e586
2022-12-01T11:22:37.5721874Z deleted: sha256:635a5f6faf92c02fc91f0a365e65bb840b74b3b7c079ee1009e6c15cd7c0e7b4
2022-12-01T11:22:37.5722635Z deleted: sha256:2696ecc06a0fc00b7ef534ef78758030c9d69f082f8fcb2f5ab275c6fc199e7c
2022-12-01T11:22:37.5723386Z deleted: sha256:cec67903bb9f9cc16c8826dff0d76c876d88147fecd2dec21ebc0adc56e4f149
2022-12-01T11:22:37.5724108Z deleted: sha256:6d95f163e75e744dc79893bc297644cf2a70187540d68c847baeb77483c6a0f5
2022-12-01T11:22:37.5724815Z deleted: sha256:1b1204e913bbc6e599fc1cdcabede5cd1315dc1b55cd67052bc665a8f8e3cc1a
2022-12-01T11:22:37.5725546Z deleted: sha256:93670f5e92be335224cf3a8b69a526306d2fec5e5e4e42c272b8c0458060036a
2022-12-01T11:22:37.5726237Z deleted: sha256:a0b1388f987804a153c13745107eb2a30de9cc6dc9eab87ae191707765dd7175
2022-12-01T11:22:37.5726916Z deleted: sha256:129bdb873e79117f4e90135f0c6a58f775fcf596f4eb514b803771cef2da8278
2022-12-01T11:22:37.5727671Z deleted: sha256:2d49e3a81bd436bfd20fb4a849cdc98da82cb74afef3de38dda7a946d3fc4153
2022-12-01T11:22:37.5728426Z deleted: sha256:0ba4e259108e5311ddf6b79ae3a35f8f16a4004ef8817e50427baa3cc90ac081
2022-12-01T11:22:37.5729269Z deleted: sha256:c164403226561914f16becdeca65c54d20dba8dad414b062efc34c05c47bf725
2022-12-01T11:22:37.5730002Z deleted: sha256:cbe4006b2e6286d50c1b292fb71b69d5299d65f055285519eafc41eac3ef8a3c
2022-12-01T11:22:37.5730753Z deleted: sha256:edcec18dceb25f1a03ec20de4676464613e69072875a83f5c45e45a31aafc5b9
2022-12-01T11:22:37.5731513Z deleted: sha256:13c4f317ac4bb48997302756b8d5f8b602e835607c9806a1a5b200e9a0657d8a
2022-12-01T11:22:37.5732183Z deleted: sha256:57f043e380f4586c76968d6e062b50bac55254a5be7e80bea3c027a5bb316469
2022-12-01T11:22:37.5732911Z deleted: sha256:3e549931e0240b9aac25dc79ed6a6259863879a5c9bd20755f77cac27c1ab8c8
2022-12-01T11:22:37.5733300Z 
2022-12-01T11:22:37.5808251Z Total reclaimed space: 18.87GB
2022-12-01T11:22:37.5871457Z Post job cleanup.
2022-12-01T11:22:37.5911003Z Post job cleanup.
2022-12-01T11:22:37.7304034Z [command]/usr/bin/git version
2022-12-01T11:22:37.7360634Z git version 2.37.1
2022-12-01T11:22:37.7426585Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4675074a-1905-4711-a20d-f93b325a0a63' before making global git config changes
2022-12-01T11:22:37.7427172Z Adding repository directory to the temporary git global config as a safe directory
2022-12-01T11:22:37.7433783Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T11:22:37.7478887Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-12-01T11:22:37.7516654Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-12-01T11:22:37.7848706Z Entering 'android/libs/fbjni'
2022-12-01T11:22:37.7892255Z Entering 'third_party/FP16'
2022-12-01T11:22:37.7937365Z Entering 'third_party/FXdiv'
2022-12-01T11:22:37.7980285Z Entering 'third_party/NNPACK'
2022-12-01T11:22:37.8023974Z Entering 'third_party/QNNPACK'
2022-12-01T11:22:37.8068469Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T11:22:37.8111838Z Entering 'third_party/XNNPACK'
2022-12-01T11:22:37.8166555Z Entering 'third_party/benchmark'
2022-12-01T11:22:37.8210373Z Entering 'third_party/cpuinfo'
2022-12-01T11:22:37.8255882Z Entering 'third_party/cub'
2022-12-01T11:22:37.8299503Z Entering 'third_party/cudnn_frontend'
2022-12-01T11:22:37.8347921Z Entering 'third_party/cutlass'
2022-12-01T11:22:37.8398786Z Entering 'third_party/eigen'
2022-12-01T11:22:37.8444470Z Entering 'third_party/fbgemm'
2022-12-01T11:22:37.8488669Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T11:22:37.8532567Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T11:22:37.8575913Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T11:22:37.8618767Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T11:22:37.8663179Z Entering 'third_party/flatbuffers'
2022-12-01T11:22:37.8709388Z Entering 'third_party/fmt'
2022-12-01T11:22:37.8752188Z Entering 'third_party/foxi'
2022-12-01T11:22:37.8795480Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T11:22:37.8841141Z Entering 'third_party/gloo'
2022-12-01T11:22:37.8886472Z Entering 'third_party/googletest'
2022-12-01T11:22:37.8929317Z Entering 'third_party/ideep'
2022-12-01T11:22:37.8972280Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T11:22:37.9016936Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T11:22:37.9066463Z Entering 'third_party/ios-cmake'
2022-12-01T11:22:37.9109259Z Entering 'third_party/ittapi'
2022-12-01T11:22:37.9151458Z Entering 'third_party/kineto'
2022-12-01T11:22:37.9194350Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T11:22:37.9236833Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T11:22:37.9281453Z Entering 'third_party/nccl/nccl'
2022-12-01T11:22:37.9325323Z Entering 'third_party/neon2sse'
2022-12-01T11:22:37.9367367Z Entering 'third_party/nlohmann'
2022-12-01T11:22:37.9412049Z Entering 'third_party/onnx'
2022-12-01T11:22:37.9471371Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T11:22:37.9513999Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T11:22:37.9559957Z Entering 'third_party/onnx-tensorrt'
2022-12-01T11:22:37.9602990Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T11:22:37.9651768Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T11:22:37.9697003Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T11:22:37.9740609Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T11:22:37.9789365Z Entering 'third_party/pocketfft'
2022-12-01T11:22:37.9832664Z Entering 'third_party/protobuf'
2022-12-01T11:22:37.9880114Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T11:22:37.9924235Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T11:22:37.9969149Z Entering 'third_party/psimd'
2022-12-01T11:22:38.0013283Z Entering 'third_party/pthreadpool'
2022-12-01T11:22:38.0057627Z Entering 'third_party/pybind11'
2022-12-01T11:22:38.0101031Z Entering 'third_party/python-enum'
2022-12-01T11:22:38.0145236Z Entering 'third_party/python-peachpy'
2022-12-01T11:22:38.0187848Z Entering 'third_party/python-six'
2022-12-01T11:22:38.0231099Z Entering 'third_party/sleef'
2022-12-01T11:22:38.0274193Z Entering 'third_party/tbb'
2022-12-01T11:22:38.0319831Z Entering 'third_party/tensorpipe'
2022-12-01T11:22:38.0363833Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T11:22:38.0406760Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T11:22:38.0449180Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T11:22:38.0491996Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T11:22:38.0534675Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T11:22:38.0580214Z Entering 'third_party/zstd'
2022-12-01T11:22:38.0641322Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-12-01T11:22:38.0672902Z http.https://github.com/.extraheader
2022-12-01T11:22:38.0682772Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-12-01T11:22:38.0720444Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-12-01T11:22:38.1045804Z Entering 'android/libs/fbjni'
2022-12-01T11:22:38.1071703Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1105057Z Entering 'third_party/FP16'
2022-12-01T11:22:38.1130181Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1164139Z Entering 'third_party/FXdiv'
2022-12-01T11:22:38.1190840Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1224694Z Entering 'third_party/NNPACK'
2022-12-01T11:22:38.1249807Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1283236Z Entering 'third_party/QNNPACK'
2022-12-01T11:22:38.1309041Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1342428Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T11:22:38.1367877Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1401027Z Entering 'third_party/XNNPACK'
2022-12-01T11:22:38.1427904Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1472048Z Entering 'third_party/benchmark'
2022-12-01T11:22:38.1497620Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1530590Z Entering 'third_party/cpuinfo'
2022-12-01T11:22:38.1555649Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1588766Z Entering 'third_party/cub'
2022-12-01T11:22:38.1613918Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1648718Z Entering 'third_party/cudnn_frontend'
2022-12-01T11:22:38.1675376Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1714868Z Entering 'third_party/cutlass'
2022-12-01T11:22:38.1740831Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1781549Z Entering 'third_party/eigen'
2022-12-01T11:22:38.1806695Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1842841Z Entering 'third_party/fbgemm'
2022-12-01T11:22:38.1868418Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1901559Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T11:22:38.1925791Z http.https://github.com/.extraheader
2022-12-01T11:22:38.1959046Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T11:22:38.1984080Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2018018Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T11:22:38.2043147Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2076088Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T11:22:38.2102323Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2136350Z Entering 'third_party/flatbuffers'
2022-12-01T11:22:38.2161785Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2197377Z Entering 'third_party/fmt'
2022-12-01T11:22:38.2223052Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2256179Z Entering 'third_party/foxi'
2022-12-01T11:22:38.2280998Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2314491Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T11:22:38.2340460Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2373179Z Entering 'third_party/gloo'
2022-12-01T11:22:38.2398772Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2431354Z Entering 'third_party/googletest'
2022-12-01T11:22:38.2457321Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2490857Z Entering 'third_party/ideep'
2022-12-01T11:22:38.2516567Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2548901Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T11:22:38.2573082Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2608271Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T11:22:38.2633241Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2672781Z Entering 'third_party/ios-cmake'
2022-12-01T11:22:38.2699738Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2732432Z Entering 'third_party/ittapi'
2022-12-01T11:22:38.2758501Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2791652Z Entering 'third_party/kineto'
2022-12-01T11:22:38.2817153Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2849792Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T11:22:38.2875499Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2908957Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T11:22:38.2933357Z http.https://github.com/.extraheader
2022-12-01T11:22:38.2968275Z Entering 'third_party/nccl/nccl'
2022-12-01T11:22:38.2995456Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3029142Z Entering 'third_party/neon2sse'
2022-12-01T11:22:38.3054698Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3087237Z Entering 'third_party/nlohmann'
2022-12-01T11:22:38.3113704Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3149462Z Entering 'third_party/onnx'
2022-12-01T11:22:38.3174393Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3221849Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T11:22:38.3246651Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3280479Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T11:22:38.3305703Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3341648Z Entering 'third_party/onnx-tensorrt'
2022-12-01T11:22:38.3366259Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3398624Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T11:22:38.3424695Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3463031Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T11:22:38.3488187Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3522595Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T11:22:38.3548656Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3581780Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T11:22:38.3607131Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3646216Z Entering 'third_party/pocketfft'
2022-12-01T11:22:38.3671904Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3706192Z Entering 'third_party/protobuf'
2022-12-01T11:22:38.3730893Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3767981Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T11:22:38.3793533Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3826979Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T11:22:38.3850993Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3886344Z Entering 'third_party/psimd'
2022-12-01T11:22:38.3911745Z http.https://github.com/.extraheader
2022-12-01T11:22:38.3944288Z Entering 'third_party/pthreadpool'
2022-12-01T11:22:38.3969071Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4002500Z Entering 'third_party/pybind11'
2022-12-01T11:22:38.4027867Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4061010Z Entering 'third_party/python-enum'
2022-12-01T11:22:38.4085801Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4118343Z Entering 'third_party/python-peachpy'
2022-12-01T11:22:38.4143575Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4176397Z Entering 'third_party/python-six'
2022-12-01T11:22:38.4201017Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4234097Z Entering 'third_party/sleef'
2022-12-01T11:22:38.4259242Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4292541Z Entering 'third_party/tbb'
2022-12-01T11:22:38.4317618Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4352527Z Entering 'third_party/tensorpipe'
2022-12-01T11:22:38.4378019Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4411189Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T11:22:38.4436796Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4470695Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T11:22:38.4497065Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4531761Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T11:22:38.4557766Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4591290Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T11:22:38.4617461Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4650157Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T11:22:38.4675290Z http.https://github.com/.extraheader
2022-12-01T11:22:38.4712353Z Entering 'third_party/zstd'
2022-12-01T11:22:38.4737268Z http.https://github.com/.extraheader
2022-12-01T11:22:38.5046567Z Cleaning up orphan processes