2023-01-11T21:02:11.5151799Z Requested labels: linux.8xlarge.nvidia.gpu
2023-01-11T21:02:11.5151897Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge
2023-01-11T21:02:11.5152051Z Reusable workflow chain:
2023-01-11T21:02:11.5152091Z pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c)
2023-01-11T21:02:11.5152137Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c)
2023-01-11T21:02:11.5152165Z Waiting for a runner to pick up this job...
2023-01-11T21:02:11.7300735Z Job is about to start running on the runner: i-03a86a33898d1fa3b (organization)
2023-01-11T21:02:16.4414904Z Current runner version: '2.300.2'
2023-01-11T21:02:16.4422650Z Runner name: 'i-03a86a33898d1fa3b'
2023-01-11T21:02:16.4423373Z Runner group name: 'Default'
2023-01-11T21:02:16.4424117Z Machine name: 'ip-10-0-3-22'
2023-01-11T21:02:16.4426912Z ##[group]GITHUB_TOKEN Permissions
2023-01-11T21:02:16.4427791Z Actions: read
2023-01-11T21:02:16.4428231Z Checks: read
2023-01-11T21:02:16.4428620Z Contents: read
2023-01-11T21:02:16.4429156Z Deployments: read
2023-01-11T21:02:16.4429608Z Discussions: read
2023-01-11T21:02:16.4429987Z Issues: read
2023-01-11T21:02:16.4430419Z Metadata: read
2023-01-11T21:02:16.4430896Z Packages: read
2023-01-11T21:02:16.4431276Z Pages: read
2023-01-11T21:02:16.4431708Z PullRequests: read
2023-01-11T21:02:16.4432206Z RepositoryProjects: read
2023-01-11T21:02:16.4432635Z SecurityEvents: read
2023-01-11T21:02:16.4433132Z Statuses: read
2023-01-11T21:02:16.4433584Z ##[endgroup]
2023-01-11T21:02:16.4437746Z Secret source: None
2023-01-11T21:02:16.4438658Z Prepare workflow directory
2023-01-11T21:02:16.5761941Z Prepare all required actions
2023-01-11T21:02:16.5987046Z Getting action download info
2023-01-11T21:02:16.8437188Z Download action repository 'pytorch/test-infra@main' (SHA:2c225610d00fb13c04fcd60389d3e4d8326167c3)
2023-01-11T21:02:17.2793968Z Download action repository 'pytorch/pytorch@master' (SHA:c5836153f5332ca83d5cacde38f2829a4d54793e)
2023-01-11T21:02:20.8443901Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2023-01-11T21:02:21.5450609Z Getting action download info
2023-01-11T21:02:21.8354605Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2023-01-11T21:02:22.0257576Z Getting action download info
2023-01-11T21:02:22.2211421Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2023-01-11T21:02:22.3821771Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2023-01-11T21:02:22.3824344Z ##[group] Inputs
2023-01-11T21:02:22.3824759Z   build-environment: linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T21:02:22.3826022Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "deploy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
]}

2023-01-11T21:02:22.3827380Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:02:22.3827884Z   sync-tag: 
2023-01-11T21:02:22.3828980Z   timeout-minutes: 240
2023-01-11T21:02:22.3829259Z   use-gha: 
2023-01-11T21:02:22.3829530Z ##[endgroup]
2023-01-11T21:02:22.3830380Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu)
2023-01-11T21:02:22.4931117Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2023-01-11T21:02:22.4931512Z with:
2023-01-11T21:02:22.4932087Z   github-secret: ***
2023-01-11T21:02:22.4932564Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2023-01-11T21:02:22.4933017Z   activate-with-label: false
2023-01-11T21:02:22.4933269Z   label: with-ssh
2023-01-11T21:02:22.4933538Z   remove-existing-keys: true
2023-01-11T21:02:22.4933802Z env:
2023-01-11T21:02:22.4934030Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:02:22.4934467Z ##[endgroup]
2023-01-11T21:02:23.0101479Z Grabbing public ssh keys from https://github.com/LucaLumetti.keys
2023-01-11T21:02:23.1130293Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2023-01-11T21:02:23.1151820Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2023-01-11T21:02:23.1200243Z Login using: ssh ec2-user@ec2-3-91-69-71.compute-1.amazonaws.com
2023-01-11T21:02:23.1200693Z All testing is done inside the container, to start an interactive session run:
2023-01-11T21:02:23.1201159Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2023-01-11T21:02:23.1496055Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2023-01-11T21:02:23.1496412Z with:
2023-01-11T21:02:23.1496657Z   submodules: recursive
2023-01-11T21:02:23.1496912Z   fetch-depth: 0
2023-01-11T21:02:23.1497126Z env:
2023-01-11T21:02:23.1497363Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:02:23.1497616Z ##[endgroup]
2023-01-11T21:02:23.1797868Z ##[group]Run retry () {
2023-01-11T21:02:23.1798194Z retry () {
2023-01-11T21:02:23.1798487Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2023-01-11T21:02:23.1798783Z }
2023-01-11T21:02:23.1799043Z echo "${GITHUB_WORKSPACE}"
2023-01-11T21:02:23.1799320Z if [ -z "${NO_SUDO}" ]; then
2023-01-11T21:02:23.1799902Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T21:02:23.1800187Z else
2023-01-11T21:02:23.1800463Z   retry rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T21:02:23.1800712Z fi
2023-01-11T21:02:23.1801011Z mkdir "${GITHUB_WORKSPACE}"
2023-01-11T21:02:23.1819335Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:02:23.1819645Z env:
2023-01-11T21:02:23.1819897Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:02:23.1820155Z   NO_SUDO: 
2023-01-11T21:02:23.1820377Z ##[endgroup]
2023-01-11T21:02:23.1947423Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:02:23.2393694Z ##[group]Run malfet/checkout@silent-checkout
2023-01-11T21:02:23.2393971Z with:
2023-01-11T21:02:23.2394251Z   ref: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:02:23.2394543Z   fetch-depth: 0
2023-01-11T21:02:23.2394778Z   submodules: recursive
2023-01-11T21:02:23.2395038Z   quiet-checkout: true
2023-01-11T21:02:23.2395318Z   repository: pytorch/pytorch
2023-01-11T21:02:23.2395725Z   token: ***
2023-01-11T21:02:23.2395979Z   ssh-strict: true
2023-01-11T21:02:23.2396248Z   persist-credentials: true
2023-01-11T21:02:23.2396512Z   clean: true
2023-01-11T21:02:23.2396730Z   lfs: false
2023-01-11T21:02:23.2396985Z   set-safe-directory: true
2023-01-11T21:02:23.2397233Z env:
2023-01-11T21:02:23.2397452Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:02:23.2397710Z ##[endgroup]
2023-01-11T21:02:23.3895841Z Syncing repository: pytorch/pytorch
2023-01-11T21:02:23.3897922Z ##[group]Getting Git version info
2023-01-11T21:02:23.3898505Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T21:02:23.3899243Z [command]/usr/bin/git version
2023-01-11T21:02:23.3899588Z git version 2.38.1
2023-01-11T21:02:23.3904546Z ##[endgroup]
2023-01-11T21:02:23.3923857Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/02a69d0e-d12b-4606-9eb1-63c71bf3a56f' before making global git config changes
2023-01-11T21:02:23.3925307Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T21:02:23.3931533Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:02:23.3976552Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T21:02:23.3983344Z ##[group]Initializing the repository
2023-01-11T21:02:23.3987024Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:02:23.4018558Z hint: Using 'master' as the name for the initial branch. This default branch name
2023-01-11T21:02:23.4020011Z hint: is subject to change. To configure the initial branch name to use in all
2023-01-11T21:02:23.4020578Z hint: of your new repositories, which will suppress this warning, call:
2023-01-11T21:02:23.4020929Z hint: 
2023-01-11T21:02:23.4021405Z hint: 	git config --global init.defaultBranch <name>
2023-01-11T21:02:23.4021770Z hint: 
2023-01-11T21:02:23.4022688Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2023-01-11T21:02:23.4023816Z hint: 'development'. The just-created branch can be renamed via this command:
2023-01-11T21:02:23.4024218Z hint: 
2023-01-11T21:02:23.4024717Z hint: 	git branch -m <name>
2023-01-11T21:02:23.4025267Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2023-01-11T21:02:23.4033334Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2023-01-11T21:02:23.4066679Z ##[endgroup]
2023-01-11T21:02:23.4067683Z ##[group]Disabling automatic garbage collection
2023-01-11T21:02:23.4071556Z [command]/usr/bin/git config --local gc.auto 0
2023-01-11T21:02:23.4102538Z ##[endgroup]
2023-01-11T21:02:23.4103654Z ##[group]Setting up auth
2023-01-11T21:02:23.4112930Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T21:02:23.4147069Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T21:02:23.4466937Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T21:02:23.4498447Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T21:02:23.4791563Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T21:02:23.4838400Z ##[endgroup]
2023-01-11T21:02:23.4839351Z ##[group]Fetching the repository
2023-01-11T21:02:23.4848332Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2023-01-11T21:03:20.3073087Z [command]/usr/bin/git rev-parse --verify --quiet 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e^{object}
2023-01-11T21:03:20.3100097Z 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:03:20.3106412Z ##[endgroup]
2023-01-11T21:03:20.3107143Z ##[group]Determining the checkout info
2023-01-11T21:03:20.3108318Z ##[endgroup]
2023-01-11T21:03:20.3109137Z ##[group]Checking out the ref
2023-01-11T21:03:20.3113614Z [command]/usr/bin/git checkout --quiet --force 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:03:22.0604913Z ##[endgroup]
2023-01-11T21:03:22.0605726Z ##[group]Setting up auth for fetching submodules
2023-01-11T21:03:22.0611276Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T21:03:22.0666960Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2023-01-11T21:03:22.0699162Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2023-01-11T21:03:22.0730476Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2023-01-11T21:03:22.0760836Z ##[endgroup]
2023-01-11T21:03:22.0761413Z ##[group]Fetching submodules
2023-01-11T21:03:22.0766341Z [command]/usr/bin/git submodule sync --recursive
2023-01-11T21:03:22.1087692Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2023-01-11T21:03:22.1391431Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2023-01-11T21:03:22.1394693Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2023-01-11T21:03:22.1398128Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2023-01-11T21:03:22.1402597Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2023-01-11T21:03:22.1406261Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2023-01-11T21:03:22.1410487Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2023-01-11T21:03:22.1414550Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2023-01-11T21:03:22.1418840Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2023-01-11T21:03:22.1423276Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2023-01-11T21:03:22.1427829Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2023-01-11T21:03:22.1432669Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2023-01-11T21:03:22.1437364Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2023-01-11T21:03:22.1442963Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2023-01-11T21:03:22.1448219Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2023-01-11T21:03:22.1453325Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2023-01-11T21:03:22.1458687Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2023-01-11T21:03:22.1464197Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2023-01-11T21:03:22.1469947Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:03:22.1475610Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2023-01-11T21:03:22.1483653Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2023-01-11T21:03:22.1489694Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2023-01-11T21:03:22.1495980Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2023-01-11T21:03:22.1502357Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2023-01-11T21:03:22.1508919Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2023-01-11T21:03:22.1515447Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2023-01-11T21:03:22.1522729Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2023-01-11T21:03:22.1529624Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2023-01-11T21:03:22.1536540Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2023-01-11T21:03:22.1543762Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2023-01-11T21:03:22.1550937Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2023-01-11T21:03:22.1558348Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2023-01-11T21:03:22.1566397Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2023-01-11T21:03:22.1574090Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2023-01-11T21:03:22.1581838Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2023-01-11T21:03:22.1589851Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2023-01-11T21:03:22.1597841Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2023-01-11T21:03:22.1607523Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2023-01-11T21:03:22.1615709Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2023-01-11T21:03:22.1624330Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2023-01-11T21:03:22.1632923Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2023-01-11T21:03:22.1642009Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2023-01-11T21:03:22.1671617Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2023-01-11T21:03:22.5446878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2023-01-11T21:03:22.7797917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2023-01-11T21:03:22.9918743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2023-01-11T21:03:23.2961514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2023-01-11T21:03:23.6487401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2023-01-11T21:03:25.8588824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2023-01-11T21:03:32.1670911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2023-01-11T21:03:32.6434133Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2023-01-11T21:03:33.2994332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2023-01-11T21:03:34.9819189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2023-01-11T21:03:36.4527301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2023-01-11T21:03:38.0564883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2023-01-11T21:03:45.3429481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2023-01-11T21:03:46.2186710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2023-01-11T21:03:47.9157326Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2023-01-11T21:03:49.1839892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2023-01-11T21:03:49.3957108Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2023-01-11T21:03:49.8975374Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2023-01-11T21:03:50.2508808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2023-01-11T21:03:51.2936289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2023-01-11T21:03:51.9349514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2023-01-11T21:03:52.1657950Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2023-01-11T21:03:52.4654848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2023-01-11T21:03:55.2548969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2023-01-11T21:03:55.6605713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2023-01-11T21:03:56.0856681Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2023-01-11T21:04:02.7586475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2023-01-11T21:04:05.4522189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2023-01-11T21:04:05.9709458Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2023-01-11T21:04:07.4183204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2023-01-11T21:04:14.4702452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2023-01-11T21:04:14.7076877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2023-01-11T21:04:14.9555709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2023-01-11T21:04:16.4703027Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2023-01-11T21:04:16.7112363Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2023-01-11T21:04:17.0673961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2023-01-11T21:04:17.4524592Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2023-01-11T21:04:18.0574012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2023-01-11T21:04:20.4523819Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2023-01-11T21:04:21.0361778Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2023-01-11T21:04:24.2338710Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2023-01-11T21:04:24.2464230Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2023-01-11T21:04:24.2562347Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2023-01-11T21:04:24.2843313Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2023-01-11T21:04:24.3114338Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2023-01-11T21:04:24.3537608Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2023-01-11T21:04:25.1138979Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2023-01-11T21:04:25.1383173Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T21:04:25.2603882Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2023-01-11T21:04:25.3001497Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2023-01-11T21:04:25.6548524Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2023-01-11T21:04:26.1727795Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2023-01-11T21:04:26.4847774Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2023-01-11T21:04:26.5416441Z Submodule path 'third_party/fbgemm': checked out '80d64206c07879fd4683be66873de7cefa1a0a71'
2023-01-11T21:04:26.5434870Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:04:26.5437858Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:04:26.5441607Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:04:26.5445306Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:04:26.5472510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2023-01-11T21:04:27.6021091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2023-01-11T21:04:28.2416719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2023-01-11T21:04:29.3234492Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2023-01-11T21:04:29.7164641Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2023-01-11T21:04:29.8399311Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2023-01-11T21:04:29.9090747Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2023-01-11T21:04:29.9205768Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2023-01-11T21:04:30.0252575Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2023-01-11T21:04:30.0673816Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2023-01-11T21:04:30.0775052Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2023-01-11T21:04:30.1250139Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2023-01-11T21:04:30.1532045Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2023-01-11T21:04:30.2068997Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2023-01-11T21:04:30.2202886Z Submodule path 'third_party/ideep': checked out 'e533c771a1e75a1c225c14b2261eefa62681d9e6'
2023-01-11T21:04:30.2219905Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2023-01-11T21:04:30.2246633Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2023-01-11T21:04:39.3488782Z Submodule path 'third_party/ideep/mkl-dnn': checked out '404ad76ee633c939d705eb583ffe50a806969d5e'
2023-01-11T21:04:39.3509253Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:04:39.3537732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2023-01-11T21:04:48.9762245Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'fbec3e25a559ee252022ae066817b204e106a6ba'
2023-01-11T21:04:48.9878222Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2023-01-11T21:04:49.0049652Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2023-01-11T21:04:49.1155629Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2023-01-11T21:04:49.1173090Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:04:49.1176264Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:04:49.1202890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2023-01-11T21:04:50.3602208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2023-01-11T21:04:51.4600550Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2023-01-11T21:04:51.5235821Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2023-01-11T21:04:51.5490968Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2023-01-11T21:04:51.5644547Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2023-01-11T21:04:51.6956754Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2023-01-11T21:04:52.0058010Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2023-01-11T21:04:52.0089643Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2023-01-11T21:04:52.0092760Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2023-01-11T21:04:52.0120603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2023-01-11T21:04:52.5033662Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2023-01-11T21:04:53.4649363Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T21:04:53.5023272Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2023-01-11T21:04:53.5196704Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2023-01-11T21:04:53.5213451Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:04:53.5240318Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2023-01-11T21:04:55.5233197Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2023-01-11T21:04:55.5256053Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:04:55.5259050Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:04:55.5287235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2023-01-11T21:04:55.9950152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2023-01-11T21:04:57.5824399Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2023-01-11T21:04:57.6628103Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2023-01-11T21:04:57.6645476Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:04:57.6672839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2023-01-11T21:04:58.7954402Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T21:04:58.8057919Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2023-01-11T21:04:59.1229327Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2023-01-11T21:04:59.1252658Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:04:59.1255802Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2023-01-11T21:04:59.1283461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2023-01-11T21:04:59.6132327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2023-01-11T21:05:00.7225304Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2023-01-11T21:05:00.8047485Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2023-01-11T21:05:00.8141430Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2023-01-11T21:05:00.8265078Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2023-01-11T21:05:00.8656433Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2023-01-11T21:05:00.8755653Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2023-01-11T21:05:00.9086907Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2023-01-11T21:05:00.9189781Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2023-01-11T21:05:00.9711369Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2023-01-11T21:05:01.1072467Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2023-01-11T21:05:01.1382165Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2023-01-11T21:05:01.1400591Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:05:01.1404293Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:05:01.1407707Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:05:01.1411276Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:05:01.1437415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2023-01-11T21:05:02.5576985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2023-01-11T21:05:02.9495471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2023-01-11T21:05:04.4416685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2023-01-11T21:05:05.5751402Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2023-01-11T21:05:05.5926259Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2023-01-11T21:05:05.6710556Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2023-01-11T21:05:05.7042546Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2023-01-11T21:05:05.7059552Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:05:05.7087244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2023-01-11T21:05:05.9640917Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T21:05:06.1274842Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2023-01-11T21:05:06.1305580Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2023-01-11T21:05:06.1625370Z Entering 'android/libs/fbjni'
2023-01-11T21:05:06.1669124Z Entering 'third_party/FP16'
2023-01-11T21:05:06.1712309Z Entering 'third_party/FXdiv'
2023-01-11T21:05:06.1756068Z Entering 'third_party/NNPACK'
2023-01-11T21:05:06.1799482Z Entering 'third_party/QNNPACK'
2023-01-11T21:05:06.1842904Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:05:06.1885583Z Entering 'third_party/XNNPACK'
2023-01-11T21:05:06.1939911Z Entering 'third_party/benchmark'
2023-01-11T21:05:06.1983297Z Entering 'third_party/cpuinfo'
2023-01-11T21:05:06.2026521Z Entering 'third_party/cub'
2023-01-11T21:05:06.2069848Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:05:06.2117743Z Entering 'third_party/cutlass'
2023-01-11T21:05:06.2168956Z Entering 'third_party/eigen'
2023-01-11T21:05:06.2213251Z Entering 'third_party/fbgemm'
2023-01-11T21:05:06.2254822Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:05:06.2295553Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:05:06.2338498Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:05:06.2381288Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:05:06.2425082Z Entering 'third_party/flatbuffers'
2023-01-11T21:05:06.2469574Z Entering 'third_party/fmt'
2023-01-11T21:05:06.2512585Z Entering 'third_party/foxi'
2023-01-11T21:05:06.2555699Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:05:06.2597408Z Entering 'third_party/gloo'
2023-01-11T21:05:06.2639389Z Entering 'third_party/googletest'
2023-01-11T21:05:06.2682448Z Entering 'third_party/ideep'
2023-01-11T21:05:06.2724210Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:05:06.2769339Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:05:06.2819011Z Entering 'third_party/ios-cmake'
2023-01-11T21:05:06.2860863Z Entering 'third_party/ittapi'
2023-01-11T21:05:06.2903065Z Entering 'third_party/kineto'
2023-01-11T21:05:06.2945589Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:05:06.2987746Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:05:06.3031310Z Entering 'third_party/nccl/nccl'
2023-01-11T21:05:06.3073687Z Entering 'third_party/neon2sse'
2023-01-11T21:05:06.3115644Z Entering 'third_party/nlohmann'
2023-01-11T21:05:06.3159116Z Entering 'third_party/onnx'
2023-01-11T21:05:06.3215326Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:05:06.3257403Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:05:06.3302570Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:05:06.3344069Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:05:06.3390929Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:05:06.3432870Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:05:06.3474474Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:05:06.3521229Z Entering 'third_party/pocketfft'
2023-01-11T21:05:06.3564218Z Entering 'third_party/protobuf'
2023-01-11T21:05:06.3609596Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:05:06.3650904Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:05:06.3694135Z Entering 'third_party/psimd'
2023-01-11T21:05:06.3736368Z Entering 'third_party/pthreadpool'
2023-01-11T21:05:06.3778394Z Entering 'third_party/pybind11'
2023-01-11T21:05:06.3822902Z Entering 'third_party/python-enum'
2023-01-11T21:05:06.3864263Z Entering 'third_party/python-peachpy'
2023-01-11T21:05:06.3907302Z Entering 'third_party/python-six'
2023-01-11T21:05:06.3949224Z Entering 'third_party/sleef'
2023-01-11T21:05:06.3991439Z Entering 'third_party/tbb'
2023-01-11T21:05:06.4036107Z Entering 'third_party/tensorpipe'
2023-01-11T21:05:06.4077781Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:05:06.4119142Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:05:06.4160957Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:05:06.4203083Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:05:06.4244037Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:05:06.4289400Z Entering 'third_party/zstd'
2023-01-11T21:05:06.4343261Z ##[endgroup]
2023-01-11T21:05:06.4343731Z ##[group]Persisting credentials for submodules
2023-01-11T21:05:06.4351029Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2023-01-11T21:05:06.4660472Z Entering 'android/libs/fbjni'
2023-01-11T21:05:06.4701887Z Entering 'third_party/FP16'
2023-01-11T21:05:06.4743889Z Entering 'third_party/FXdiv'
2023-01-11T21:05:06.4785688Z Entering 'third_party/NNPACK'
2023-01-11T21:05:06.4827584Z Entering 'third_party/QNNPACK'
2023-01-11T21:05:06.4869290Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:05:06.4911186Z Entering 'third_party/XNNPACK'
2023-01-11T21:05:06.4963934Z Entering 'third_party/benchmark'
2023-01-11T21:05:06.5006049Z Entering 'third_party/cpuinfo'
2023-01-11T21:05:06.5048215Z Entering 'third_party/cub'
2023-01-11T21:05:06.5090474Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:05:06.5138778Z Entering 'third_party/cutlass'
2023-01-11T21:05:06.5186932Z Entering 'third_party/eigen'
2023-01-11T21:05:06.5231430Z Entering 'third_party/fbgemm'
2023-01-11T21:05:06.5273006Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:05:06.5313790Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:05:06.5354598Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:05:06.5395275Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:05:06.5436936Z Entering 'third_party/flatbuffers'
2023-01-11T21:05:06.5480157Z Entering 'third_party/fmt'
2023-01-11T21:05:06.5521723Z Entering 'third_party/foxi'
2023-01-11T21:05:06.5563952Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:05:06.5605684Z Entering 'third_party/gloo'
2023-01-11T21:05:06.5647736Z Entering 'third_party/googletest'
2023-01-11T21:05:06.5689278Z Entering 'third_party/ideep'
2023-01-11T21:05:06.5730392Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:05:06.5773362Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:05:06.5821748Z Entering 'third_party/ios-cmake'
2023-01-11T21:05:06.5863412Z Entering 'third_party/ittapi'
2023-01-11T21:05:06.5905219Z Entering 'third_party/kineto'
2023-01-11T21:05:06.5946544Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:05:06.5987829Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:05:06.6030646Z Entering 'third_party/nccl/nccl'
2023-01-11T21:05:06.6072976Z Entering 'third_party/neon2sse'
2023-01-11T21:05:06.6115069Z Entering 'third_party/nlohmann'
2023-01-11T21:05:06.6158489Z Entering 'third_party/onnx'
2023-01-11T21:05:06.6212125Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:05:06.6253317Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:05:06.6296284Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:05:06.6336726Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:05:06.6382918Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:05:06.6425580Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:05:06.6467036Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:05:06.6512882Z Entering 'third_party/pocketfft'
2023-01-11T21:05:06.6554171Z Entering 'third_party/protobuf'
2023-01-11T21:05:06.6599104Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:05:06.6640654Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:05:06.6684140Z Entering 'third_party/psimd'
2023-01-11T21:05:06.6725952Z Entering 'third_party/pthreadpool'
2023-01-11T21:05:06.6767615Z Entering 'third_party/pybind11'
2023-01-11T21:05:06.6809015Z Entering 'third_party/python-enum'
2023-01-11T21:05:06.6850247Z Entering 'third_party/python-peachpy'
2023-01-11T21:05:06.6891714Z Entering 'third_party/python-six'
2023-01-11T21:05:06.6933090Z Entering 'third_party/sleef'
2023-01-11T21:05:06.6975204Z Entering 'third_party/tbb'
2023-01-11T21:05:06.7018908Z Entering 'third_party/tensorpipe'
2023-01-11T21:05:06.7060145Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:05:06.7100996Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:05:06.7141544Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:05:06.7183735Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:05:06.7224316Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:05:06.7269908Z Entering 'third_party/zstd'
2023-01-11T21:05:06.7324483Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2023-01-11T21:05:06.7633742Z Entering 'android/libs/fbjni'
2023-01-11T21:05:06.7673651Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2023-01-11T21:05:06.7691662Z Entering 'third_party/FP16'
2023-01-11T21:05:06.7731261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2023-01-11T21:05:06.7748788Z Entering 'third_party/FXdiv'
2023-01-11T21:05:06.7787996Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2023-01-11T21:05:06.7805628Z Entering 'third_party/NNPACK'
2023-01-11T21:05:06.7844694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2023-01-11T21:05:06.7862257Z Entering 'third_party/QNNPACK'
2023-01-11T21:05:06.7900994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2023-01-11T21:05:06.7918607Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:05:06.7958174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2023-01-11T21:05:06.7976424Z Entering 'third_party/XNNPACK'
2023-01-11T21:05:06.8015088Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2023-01-11T21:05:06.8045144Z Entering 'third_party/benchmark'
2023-01-11T21:05:06.8084489Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:05:06.8101831Z Entering 'third_party/cpuinfo'
2023-01-11T21:05:06.8140406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T21:05:06.8158271Z Entering 'third_party/cub'
2023-01-11T21:05:06.8198131Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2023-01-11T21:05:06.8216127Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:05:06.8255579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2023-01-11T21:05:06.8279065Z Entering 'third_party/cutlass'
2023-01-11T21:05:06.8318110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2023-01-11T21:05:06.8342723Z Entering 'third_party/eigen'
2023-01-11T21:05:06.8381751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2023-01-11T21:05:06.8401781Z Entering 'third_party/fbgemm'
2023-01-11T21:05:06.8441018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2023-01-11T21:05:06.8458836Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:05:06.8497922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2023-01-11T21:05:06.8515018Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:05:06.8553567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T21:05:06.8572144Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:05:06.8610856Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:05:06.8628029Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:05:06.8666255Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2023-01-11T21:05:06.8684747Z Entering 'third_party/flatbuffers'
2023-01-11T21:05:06.8723584Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2023-01-11T21:05:06.8743000Z Entering 'third_party/fmt'
2023-01-11T21:05:06.8781766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2023-01-11T21:05:06.8799334Z Entering 'third_party/foxi'
2023-01-11T21:05:06.8837973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2023-01-11T21:05:06.8855710Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:05:06.8893685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2023-01-11T21:05:06.8911148Z Entering 'third_party/gloo'
2023-01-11T21:05:06.8950283Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2023-01-11T21:05:06.8968374Z Entering 'third_party/googletest'
2023-01-11T21:05:06.9007054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:05:06.9024221Z Entering 'third_party/ideep'
2023-01-11T21:05:06.9062614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2023-01-11T21:05:06.9079169Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:05:06.9117317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2023-01-11T21:05:06.9137278Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:05:06.9175591Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2023-01-11T21:05:06.9199428Z Entering 'third_party/ios-cmake'
2023-01-11T21:05:06.9238183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2023-01-11T21:05:06.9255733Z Entering 'third_party/ittapi'
2023-01-11T21:05:06.9293932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2023-01-11T21:05:06.9310699Z Entering 'third_party/kineto'
2023-01-11T21:05:06.9349813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2023-01-11T21:05:06.9367471Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:05:06.9405876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2023-01-11T21:05:06.9422838Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:05:06.9461434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2023-01-11T21:05:06.9480117Z Entering 'third_party/nccl/nccl'
2023-01-11T21:05:06.9519733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2023-01-11T21:05:06.9537547Z Entering 'third_party/neon2sse'
2023-01-11T21:05:06.9576106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2023-01-11T21:05:06.9593011Z Entering 'third_party/nlohmann'
2023-01-11T21:05:06.9632102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2023-01-11T21:05:06.9651260Z Entering 'third_party/onnx'
2023-01-11T21:05:06.9690144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2023-01-11T21:05:06.9719337Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:05:06.9757446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:05:06.9775855Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:05:06.9815957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:05:06.9835881Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:05:06.9875100Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2023-01-11T21:05:06.9892493Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:05:06.9930381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2023-01-11T21:05:06.9952406Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:05:06.9991454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:05:07.0010193Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:05:07.0048995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:05:07.0065687Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:05:07.0104420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T21:05:07.0127905Z Entering 'third_party/pocketfft'
2023-01-11T21:05:07.0166161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2023-01-11T21:05:07.0183213Z Entering 'third_party/protobuf'
2023-01-11T21:05:07.0222025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2023-01-11T21:05:07.0243532Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:05:07.0283015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:05:07.0299807Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:05:07.0337885Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:05:07.0356709Z Entering 'third_party/psimd'
2023-01-11T21:05:07.0394723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2023-01-11T21:05:07.0412672Z Entering 'third_party/pthreadpool'
2023-01-11T21:05:07.0452143Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2023-01-11T21:05:07.0469451Z Entering 'third_party/pybind11'
2023-01-11T21:05:07.0508054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:05:07.0527391Z Entering 'third_party/python-enum'
2023-01-11T21:05:07.0565795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2023-01-11T21:05:07.0582746Z Entering 'third_party/python-peachpy'
2023-01-11T21:05:07.0620864Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2023-01-11T21:05:07.0637965Z Entering 'third_party/python-six'
2023-01-11T21:05:07.0676368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2023-01-11T21:05:07.0693546Z Entering 'third_party/sleef'
2023-01-11T21:05:07.0731328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2023-01-11T21:05:07.0748606Z Entering 'third_party/tbb'
2023-01-11T21:05:07.0787324Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2023-01-11T21:05:07.0806956Z Entering 'third_party/tensorpipe'
2023-01-11T21:05:07.0845858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2023-01-11T21:05:07.0863165Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:05:07.0901134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:05:07.0918090Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:05:07.0957754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2023-01-11T21:05:07.0974853Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:05:07.1013782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2023-01-11T21:05:07.1031138Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:05:07.1069177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:05:07.1086440Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:05:07.1125434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T21:05:07.1146023Z Entering 'third_party/zstd'
2023-01-11T21:05:07.1185564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2023-01-11T21:05:07.2692109Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2023-01-11T21:05:07.3008083Z Entering 'android/libs/fbjni'
2023-01-11T21:05:07.3051154Z Entering 'third_party/FP16'
2023-01-11T21:05:07.3094624Z Entering 'third_party/FXdiv'
2023-01-11T21:05:07.3138072Z Entering 'third_party/NNPACK'
2023-01-11T21:05:07.3180785Z Entering 'third_party/QNNPACK'
2023-01-11T21:05:07.3223680Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:05:07.3267246Z Entering 'third_party/XNNPACK'
2023-01-11T21:05:07.3320440Z Entering 'third_party/benchmark'
2023-01-11T21:05:07.3363202Z Entering 'third_party/cpuinfo'
2023-01-11T21:05:07.3406563Z Entering 'third_party/cub'
2023-01-11T21:05:07.3449781Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:05:07.3500552Z Entering 'third_party/cutlass'
2023-01-11T21:05:07.3549395Z Entering 'third_party/eigen'
2023-01-11T21:05:07.3593856Z Entering 'third_party/fbgemm'
2023-01-11T21:05:07.3637230Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:05:07.3682337Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:05:07.3725512Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:05:07.3767597Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:05:07.3810703Z Entering 'third_party/flatbuffers'
2023-01-11T21:05:07.3854935Z Entering 'third_party/fmt'
2023-01-11T21:05:07.3897034Z Entering 'third_party/foxi'
2023-01-11T21:05:07.3938963Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:05:07.3980996Z Entering 'third_party/gloo'
2023-01-11T21:05:07.4023960Z Entering 'third_party/googletest'
2023-01-11T21:05:07.4066342Z Entering 'third_party/ideep'
2023-01-11T21:05:07.4107864Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:05:07.4151967Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:05:07.4202223Z Entering 'third_party/ios-cmake'
2023-01-11T21:05:07.4245700Z Entering 'third_party/ittapi'
2023-01-11T21:05:07.4289176Z Entering 'third_party/kineto'
2023-01-11T21:05:07.4333136Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:05:07.4375247Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:05:07.4418597Z Entering 'third_party/nccl/nccl'
2023-01-11T21:05:07.4461433Z Entering 'third_party/neon2sse'
2023-01-11T21:05:07.4503935Z Entering 'third_party/nlohmann'
2023-01-11T21:05:07.4548063Z Entering 'third_party/onnx'
2023-01-11T21:05:07.4602862Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:05:07.4646568Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:05:07.4690998Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:05:07.4733716Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:05:07.4779842Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:05:07.4822310Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:05:07.4864099Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:05:07.4911111Z Entering 'third_party/pocketfft'
2023-01-11T21:05:07.4953437Z Entering 'third_party/protobuf'
2023-01-11T21:05:07.4999178Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:05:07.5042002Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:05:07.5085768Z Entering 'third_party/psimd'
2023-01-11T21:05:07.5128695Z Entering 'third_party/pthreadpool'
2023-01-11T21:05:07.5170895Z Entering 'third_party/pybind11'
2023-01-11T21:05:07.5213720Z Entering 'third_party/python-enum'
2023-01-11T21:05:07.5255405Z Entering 'third_party/python-peachpy'
2023-01-11T21:05:07.5298897Z Entering 'third_party/python-six'
2023-01-11T21:05:07.5341032Z Entering 'third_party/sleef'
2023-01-11T21:05:07.5384217Z Entering 'third_party/tbb'
2023-01-11T21:05:07.5429158Z Entering 'third_party/tensorpipe'
2023-01-11T21:05:07.5472492Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:05:07.5516094Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:05:07.5557690Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:05:07.5600040Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:05:07.5642193Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:05:07.5687303Z Entering 'third_party/zstd'
2023-01-11T21:05:07.5741749Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2023-01-11T21:05:07.6049581Z Entering 'android/libs/fbjni'
2023-01-11T21:05:07.6091626Z Entering 'third_party/FP16'
2023-01-11T21:05:07.6133846Z Entering 'third_party/FXdiv'
2023-01-11T21:05:07.6176085Z Entering 'third_party/NNPACK'
2023-01-11T21:05:07.6219504Z Entering 'third_party/QNNPACK'
2023-01-11T21:05:07.6262764Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:05:07.6306259Z Entering 'third_party/XNNPACK'
2023-01-11T21:05:07.6358394Z Entering 'third_party/benchmark'
2023-01-11T21:05:07.6400685Z Entering 'third_party/cpuinfo'
2023-01-11T21:05:07.6444234Z Entering 'third_party/cub'
2023-01-11T21:05:07.6486827Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:05:07.6535185Z Entering 'third_party/cutlass'
2023-01-11T21:05:07.6585102Z Entering 'third_party/eigen'
2023-01-11T21:05:07.6630390Z Entering 'third_party/fbgemm'
2023-01-11T21:05:07.6673575Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:05:07.6715685Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:05:07.6759307Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:05:07.6801993Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:05:07.6844725Z Entering 'third_party/flatbuffers'
2023-01-11T21:05:07.6890719Z Entering 'third_party/fmt'
2023-01-11T21:05:07.6932775Z Entering 'third_party/foxi'
2023-01-11T21:05:07.6974857Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:05:07.7017701Z Entering 'third_party/gloo'
2023-01-11T21:05:07.7061143Z Entering 'third_party/googletest'
2023-01-11T21:05:07.7103569Z Entering 'third_party/ideep'
2023-01-11T21:05:07.7145858Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:05:07.7190136Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:05:07.7240076Z Entering 'third_party/ios-cmake'
2023-01-11T21:05:07.7282175Z Entering 'third_party/ittapi'
2023-01-11T21:05:07.7324113Z Entering 'third_party/kineto'
2023-01-11T21:05:07.7366145Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:05:07.7408546Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:05:07.7451501Z Entering 'third_party/nccl/nccl'
2023-01-11T21:05:07.7494257Z Entering 'third_party/neon2sse'
2023-01-11T21:05:07.7535971Z Entering 'third_party/nlohmann'
2023-01-11T21:05:07.7579199Z Entering 'third_party/onnx'
2023-01-11T21:05:07.7634655Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:05:07.7676638Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:05:07.7720859Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:05:07.7762403Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:05:07.7809125Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:05:07.7852483Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:05:07.7894997Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:05:07.7941713Z Entering 'third_party/pocketfft'
2023-01-11T21:05:07.7983702Z Entering 'third_party/protobuf'
2023-01-11T21:05:07.8030810Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:05:07.8073288Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:05:07.8117468Z Entering 'third_party/psimd'
2023-01-11T21:05:07.8160060Z Entering 'third_party/pthreadpool'
2023-01-11T21:05:07.8202819Z Entering 'third_party/pybind11'
2023-01-11T21:05:07.8245934Z Entering 'third_party/python-enum'
2023-01-11T21:05:07.8288039Z Entering 'third_party/python-peachpy'
2023-01-11T21:05:07.8330823Z Entering 'third_party/python-six'
2023-01-11T21:05:07.8372797Z Entering 'third_party/sleef'
2023-01-11T21:05:07.8414654Z Entering 'third_party/tbb'
2023-01-11T21:05:07.8459070Z Entering 'third_party/tensorpipe'
2023-01-11T21:05:07.8501573Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:05:07.8544148Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:05:07.8585665Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:05:07.8628112Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:05:07.8669200Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:05:07.8714368Z Entering 'third_party/zstd'
2023-01-11T21:05:07.8766072Z ##[endgroup]
2023-01-11T21:05:07.8807568Z [command]/usr/bin/git log -1 --format='%H'
2023-01-11T21:05:07.8835776Z '8419ddda87c8a47eacc63b54bc7ec98c1f27c26e'
2023-01-11T21:05:07.8987286Z Prepare all required actions
2023-01-11T21:05:07.9020878Z ##[group]Run ./.github/actions/setup-linux
2023-01-11T21:05:07.9021155Z env:
2023-01-11T21:05:07.9021396Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:05:07.9021641Z ##[endgroup]
2023-01-11T21:05:07.9042417Z ##[group]Run set -euo pipefail
2023-01-11T21:05:07.9042733Z set -euo pipefail
2023-01-11T21:05:07.9043041Z function get_ec2_metadata() {
2023-01-11T21:05:07.9043364Z   # Pulled from instance metadata endpoint for EC2
2023-01-11T21:05:07.9043847Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2023-01-11T21:05:07.9044264Z   category=$1
2023-01-11T21:05:07.9044602Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2023-01-11T21:05:07.9044901Z }
2023-01-11T21:05:07.9045184Z echo "ami-id: $(get_ec2_metadata ami-id)"
2023-01-11T21:05:07.9045573Z echo "instance-id: $(get_ec2_metadata instance-id)"
2023-01-11T21:05:07.9045941Z echo "instance-type: $(get_ec2_metadata instance-type)"
2023-01-11T21:05:07.9046287Z echo "system info $(uname -a)"
2023-01-11T21:05:07.9059218Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:05:07.9059503Z env:
2023-01-11T21:05:07.9059752Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:05:07.9060017Z ##[endgroup]
2023-01-11T21:05:07.9158451Z ami-id: ami-096198a0bccc6bad4
2023-01-11T21:05:07.9221509Z instance-id: i-03a86a33898d1fa3b
2023-01-11T21:05:07.9282428Z instance-type: g3.8xlarge
2023-01-11T21:05:07.9290818Z system info Linux ip-10-0-3-22.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2023-01-11T21:05:07.9311447Z ##[group]Run if systemctl is-active --quiet docker; then
2023-01-11T21:05:07.9311834Z if systemctl is-active --quiet docker; then
2023-01-11T21:05:07.9312179Z     echo "Docker daemon is running...";
2023-01-11T21:05:07.9312446Z else
2023-01-11T21:05:07.9312773Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2023-01-11T21:05:07.9313082Z fi
2023-01-11T21:05:07.9324834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:05:07.9325141Z env:
2023-01-11T21:05:07.9325392Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:05:07.9325634Z ##[endgroup]
2023-01-11T21:05:07.9376142Z Docker daemon is running...
2023-01-11T21:05:07.9396160Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T21:05:07.9396636Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T21:05:07.9397019Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:05:07.9397513Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2023-01-11T21:05:07.9397986Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2023-01-11T21:05:07.9409340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:05:07.9409640Z env:
2023-01-11T21:05:07.9409888Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:05:07.9410141Z   AWS_RETRY_MODE: standard
2023-01-11T21:05:07.9410402Z   AWS_MAX_ATTEMPTS: 5
2023-01-11T21:05:07.9410678Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T21:05:07.9410923Z ##[endgroup]
2023-01-11T21:05:08.8656937Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2023-01-11T21:05:08.8657405Z Configure a credential helper to remove this warning. See
2023-01-11T21:05:08.8657928Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2023-01-11T21:05:08.8658192Z 
2023-01-11T21:05:08.8658845Z Login Succeeded
2023-01-11T21:05:08.8757925Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:05:08.8758338Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:05:08.8758830Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:05:08.8771854Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:05:08.8772135Z env:
2023-01-11T21:05:08.8772378Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:05:08.8772643Z ##[endgroup]
2023-01-11T21:05:08.8862961Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2023-01-11T21:05:08.8863321Z with:
2023-01-11T21:05:08.8863807Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:05:08.8864279Z env:
2023-01-11T21:05:08.8864518Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:05:08.8864777Z ##[endgroup]
2023-01-11T21:05:08.8883284Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:05:08.8883715Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:05:08.8884141Z # ignore output since only exit code is used for conditional
2023-01-11T21:05:08.8884652Z # only pull docker image if it's not available locally
2023-01-11T21:05:08.8885071Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2023-01-11T21:05:08.8885603Z   retry docker pull "${DOCKER_IMAGE}"
2023-01-11T21:05:08.8885955Z fi
2023-01-11T21:05:08.8897646Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:05:08.8898017Z env:
2023-01-11T21:05:08.8898335Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:05:08.8898867Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:05:08.8899481Z ##[endgroup]
2023-01-11T21:05:09.2508364Z fd224c2e6c79d7fdec6408da598bf52bc5b201dd: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2023-01-11T21:05:09.2509379Z fb668870d8a7: Pulling fs layer
2023-01-11T21:05:09.2509846Z 3dc32ed140fb: Pulling fs layer
2023-01-11T21:05:09.2510357Z 54a1df240516: Pulling fs layer
2023-01-11T21:05:09.2510651Z cf378b3cb3c7: Pulling fs layer
2023-01-11T21:05:09.2511703Z 9b4412378859: Pulling fs layer
2023-01-11T21:05:09.2512041Z 502253a1be21: Pulling fs layer
2023-01-11T21:05:09.2512342Z 5c7dd67e5809: Pulling fs layer
2023-01-11T21:05:09.2512690Z bdfd23ed3f48: Pulling fs layer
2023-01-11T21:05:09.2513669Z aee1dd761bdd: Pulling fs layer
2023-01-11T21:05:09.2514660Z 5feda9af2542: Pulling fs layer
2023-01-11T21:05:09.2515078Z f8371ecb849a: Pulling fs layer
2023-01-11T21:05:09.2515648Z ce4a87d45645: Pulling fs layer
2023-01-11T21:05:09.2515928Z cf378b3cb3c7: Waiting
2023-01-11T21:05:09.2516414Z 39629f7269f9: Pulling fs layer
2023-01-11T21:05:09.2516887Z 87d0ffa55850: Pulling fs layer
2023-01-11T21:05:09.2517172Z 70702f8b5bc4: Pulling fs layer
2023-01-11T21:05:09.2517615Z 0c06be5c20e0: Pulling fs layer
2023-01-11T21:05:09.2518417Z b372c2a3bc3f: Pulling fs layer
2023-01-11T21:05:09.2520900Z f8371ecb849a: Waiting
2023-01-11T21:05:09.2521245Z 582d081a59fa: Pulling fs layer
2023-01-11T21:05:09.2521642Z e1c655e7ec0e: Pulling fs layer
2023-01-11T21:05:09.2521989Z 5feda9af2542: Waiting
2023-01-11T21:05:09.2522262Z 87d0ffa55850: Waiting
2023-01-11T21:05:09.2522642Z 502253a1be21: Waiting
2023-01-11T21:05:09.2522958Z b372c2a3bc3f: Waiting
2023-01-11T21:05:09.2523243Z c7726d39d806: Pulling fs layer
2023-01-11T21:05:09.2523579Z 5c7dd67e5809: Waiting
2023-01-11T21:05:09.2525208Z 1c22f2f8c01b: Pulling fs layer
2023-01-11T21:05:09.2525530Z b8f759fd0191: Pulling fs layer
2023-01-11T21:05:09.2526113Z e28e73a4bddd: Pulling fs layer
2023-01-11T21:05:09.2526555Z 90d8f9bbe048: Pulling fs layer
2023-01-11T21:05:09.2526837Z b34bd39d0461: Pulling fs layer
2023-01-11T21:05:09.2527208Z 0c06be5c20e0: Waiting
2023-01-11T21:05:09.2527533Z 2f2308643d60: Pulling fs layer
2023-01-11T21:05:09.2527806Z aee1dd761bdd: Waiting
2023-01-11T21:05:09.2528147Z 8e3432e5a569: Pulling fs layer
2023-01-11T21:05:09.2528549Z 1c22f2f8c01b: Waiting
2023-01-11T21:05:09.2528866Z c7726d39d806: Waiting
2023-01-11T21:05:09.2529131Z 9ea746919509: Pulling fs layer
2023-01-11T21:05:09.2529478Z 582d081a59fa: Waiting
2023-01-11T21:05:09.2529778Z 39629f7269f9: Waiting
2023-01-11T21:05:09.2530077Z 1a2fd7b216d7: Pulling fs layer
2023-01-11T21:05:09.2530399Z b8f759fd0191: Waiting
2023-01-11T21:05:09.2530882Z 2f2308643d60: Waiting
2023-01-11T21:05:09.2531156Z 9ea746919509: Waiting
2023-01-11T21:05:09.2531474Z 19fde6a723a0: Pulling fs layer
2023-01-11T21:05:09.2531871Z 06369252d749: Pulling fs layer
2023-01-11T21:05:09.2532165Z ea4bfeaa0fc7: Pulling fs layer
2023-01-11T21:05:09.2532540Z a1d16b6a5070: Pulling fs layer
2023-01-11T21:05:09.2532864Z b34bd39d0461: Waiting
2023-01-11T21:05:09.2533122Z ce4a87d45645: Waiting
2023-01-11T21:05:09.2533490Z f550b7ff2470: Pulling fs layer
2023-01-11T21:05:09.2533812Z 1a2fd7b216d7: Waiting
2023-01-11T21:05:09.2534085Z 12ddc57b99eb: Pulling fs layer
2023-01-11T21:05:09.2534420Z 8345085fb0a0: Pulling fs layer
2023-01-11T21:05:09.2534788Z ea4bfeaa0fc7: Waiting
2023-01-11T21:05:09.2535072Z 4cc94dbec031: Pulling fs layer
2023-01-11T21:05:09.2535424Z 19fde6a723a0: Waiting
2023-01-11T21:05:09.2535746Z 29a7c0d5fa4c: Pulling fs layer
2023-01-11T21:05:09.2536020Z 06369252d749: Waiting
2023-01-11T21:05:09.2536357Z f550b7ff2470: Waiting
2023-01-11T21:05:09.2536683Z 25571655d0e1: Pulling fs layer
2023-01-11T21:05:09.2536968Z bdf297d7f88c: Pulling fs layer
2023-01-11T21:05:09.2537303Z 4cc94dbec031: Waiting
2023-01-11T21:05:09.2537627Z 0b3950af8ae1: Pulling fs layer
2023-01-11T21:05:09.2537975Z 25571655d0e1: Waiting
2023-01-11T21:05:09.2538260Z 29a7c0d5fa4c: Waiting
2023-01-11T21:05:09.2538578Z 12ddc57b99eb: Waiting
2023-01-11T21:05:09.2538901Z 6d68f7da8baa: Pulling fs layer
2023-01-11T21:05:09.2539187Z cca768f96df4: Pulling fs layer
2023-01-11T21:05:09.2539552Z 8c3cf3d5e1c5: Pulling fs layer
2023-01-11T21:05:09.2539887Z 70702f8b5bc4: Waiting
2023-01-11T21:05:09.2540159Z 61eecfa8b34e: Pulling fs layer
2023-01-11T21:05:09.2540481Z 0b3950af8ae1: Waiting
2023-01-11T21:05:09.2540827Z 95c1ac011645: Pulling fs layer
2023-01-11T21:05:09.2541117Z 3046cc00c4ca: Pulling fs layer
2023-01-11T21:05:09.2541480Z 6d68f7da8baa: Waiting
2023-01-11T21:05:09.2541802Z 195d560d8cf6: Pulling fs layer
2023-01-11T21:05:09.2542084Z 77250abd5ca4: Pulling fs layer
2023-01-11T21:05:09.2542439Z cca768f96df4: Waiting
2023-01-11T21:05:09.2542776Z 881b24daf9c5: Pulling fs layer
2023-01-11T21:05:09.2543047Z 95c1ac011645: Waiting
2023-01-11T21:05:09.2543355Z bdfd23ed3f48: Waiting
2023-01-11T21:05:09.2543675Z 9fbf0a18619e: Pulling fs layer
2023-01-11T21:05:09.2543998Z 02048a597c22: Pulling fs layer
2023-01-11T21:05:09.2544357Z 859052a25d95: Pulling fs layer
2023-01-11T21:05:09.2544677Z 9fbf0a18619e: Waiting
2023-01-11T21:05:09.2544947Z 3e03143da3c2: Pulling fs layer
2023-01-11T21:05:09.2545272Z 02048a597c22: Waiting
2023-01-11T21:05:09.2545607Z 881b24daf9c5: Waiting
2023-01-11T21:05:09.2545863Z bdf297d7f88c: Waiting
2023-01-11T21:05:09.2546172Z 77250abd5ca4: Waiting
2023-01-11T21:05:09.2546642Z a1d16b6a5070: Waiting
2023-01-11T21:05:09.2577383Z 90d8f9bbe048: Waiting
2023-01-11T21:05:09.2577658Z 3e03143da3c2: Waiting
2023-01-11T21:05:09.2577893Z 8345085fb0a0: Waiting
2023-01-11T21:05:09.2578144Z 195d560d8cf6: Waiting
2023-01-11T21:05:09.3770618Z 3dc32ed140fb: Verifying Checksum
2023-01-11T21:05:09.3770948Z 3dc32ed140fb: Download complete
2023-01-11T21:05:09.4724124Z cf378b3cb3c7: Download complete
2023-01-11T21:05:09.5393824Z 54a1df240516: Verifying Checksum
2023-01-11T21:05:09.5394330Z 54a1df240516: Download complete
2023-01-11T21:05:09.5543341Z 9b4412378859: Verifying Checksum
2023-01-11T21:05:09.5543620Z 9b4412378859: Download complete
2023-01-11T21:05:09.5763571Z fb668870d8a7: Verifying Checksum
2023-01-11T21:05:09.5763871Z fb668870d8a7: Download complete
2023-01-11T21:05:09.6414860Z 5c7dd67e5809: Download complete
2023-01-11T21:05:09.6758277Z bdfd23ed3f48: Verifying Checksum
2023-01-11T21:05:09.6758880Z bdfd23ed3f48: Download complete
2023-01-11T21:05:09.7212510Z aee1dd761bdd: Verifying Checksum
2023-01-11T21:05:09.7212862Z aee1dd761bdd: Download complete
2023-01-11T21:05:09.8014783Z f8371ecb849a: Verifying Checksum
2023-01-11T21:05:09.8015230Z f8371ecb849a: Download complete
2023-01-11T21:05:09.9322942Z ce4a87d45645: Verifying Checksum
2023-01-11T21:05:09.9323523Z ce4a87d45645: Download complete
2023-01-11T21:05:10.3298468Z fb668870d8a7: Pull complete
2023-01-11T21:05:10.6516178Z 3dc32ed140fb: Pull complete
2023-01-11T21:05:11.2197364Z 54a1df240516: Pull complete
2023-01-11T21:05:11.3679789Z cf378b3cb3c7: Pull complete
2023-01-11T21:05:11.5007048Z 9b4412378859: Pull complete
2023-01-11T21:05:12.0763945Z 39629f7269f9: Verifying Checksum
2023-01-11T21:05:12.0764439Z 39629f7269f9: Download complete
2023-01-11T21:05:12.1591571Z 87d0ffa55850: Verifying Checksum
2023-01-11T21:05:12.1591940Z 87d0ffa55850: Download complete
2023-01-11T21:05:12.2359840Z 70702f8b5bc4: Verifying Checksum
2023-01-11T21:05:12.3244067Z 0c06be5c20e0: Verifying Checksum
2023-01-11T21:05:12.3244607Z 0c06be5c20e0: Download complete
2023-01-11T21:05:13.3729091Z b372c2a3bc3f: Verifying Checksum
2023-01-11T21:05:13.3729530Z b372c2a3bc3f: Download complete
2023-01-11T21:05:13.4771166Z 582d081a59fa: Verifying Checksum
2023-01-11T21:05:13.4771459Z 582d081a59fa: Download complete
2023-01-11T21:05:13.5566444Z e1c655e7ec0e: Verifying Checksum
2023-01-11T21:05:13.5566787Z e1c655e7ec0e: Download complete
2023-01-11T21:05:20.7603221Z 502253a1be21: Verifying Checksum
2023-01-11T21:05:20.7603695Z 502253a1be21: Download complete
2023-01-11T21:05:20.8538351Z 1c22f2f8c01b: Download complete
2023-01-11T21:05:20.9328658Z b8f759fd0191: Verifying Checksum
2023-01-11T21:05:20.9329126Z b8f759fd0191: Download complete
2023-01-11T21:05:21.0050914Z e28e73a4bddd: Verifying Checksum
2023-01-11T21:05:21.0051596Z e28e73a4bddd: Download complete
2023-01-11T21:05:21.0868778Z 90d8f9bbe048: Download complete
2023-01-11T21:05:21.1678128Z b34bd39d0461: Verifying Checksum
2023-01-11T21:05:21.1678793Z b34bd39d0461: Download complete
2023-01-11T21:05:21.2476387Z 2f2308643d60: Verifying Checksum
2023-01-11T21:05:21.2477076Z 2f2308643d60: Download complete
2023-01-11T21:05:22.1976131Z 8e3432e5a569: Verifying Checksum
2023-01-11T21:05:22.1976479Z 8e3432e5a569: Download complete
2023-01-11T21:05:22.2724635Z 9ea746919509: Download complete
2023-01-11T21:05:22.3555117Z 1a2fd7b216d7: Download complete
2023-01-11T21:05:22.4592724Z 19fde6a723a0: Verifying Checksum
2023-01-11T21:05:22.4593334Z 19fde6a723a0: Download complete
2023-01-11T21:05:22.5416121Z 06369252d749: Verifying Checksum
2023-01-11T21:05:22.5416453Z 06369252d749: Download complete
2023-01-11T21:05:22.6280086Z ea4bfeaa0fc7: Verifying Checksum
2023-01-11T21:05:22.6280755Z ea4bfeaa0fc7: Download complete
2023-01-11T21:05:24.0397162Z 5feda9af2542: Verifying Checksum
2023-01-11T21:05:24.0397524Z 5feda9af2542: Download complete
2023-01-11T21:05:24.1183099Z f550b7ff2470: Verifying Checksum
2023-01-11T21:05:24.1183412Z f550b7ff2470: Download complete
2023-01-11T21:05:24.1970442Z 12ddc57b99eb: Verifying Checksum
2023-01-11T21:05:24.1971279Z 12ddc57b99eb: Download complete
2023-01-11T21:05:24.5860153Z 8345085fb0a0: Verifying Checksum
2023-01-11T21:05:24.5860755Z 8345085fb0a0: Download complete
2023-01-11T21:05:24.6949826Z 4cc94dbec031: Verifying Checksum
2023-01-11T21:05:24.6950535Z 4cc94dbec031: Download complete
2023-01-11T21:05:24.7793953Z 29a7c0d5fa4c: Verifying Checksum
2023-01-11T21:05:24.7794776Z 29a7c0d5fa4c: Download complete
2023-01-11T21:05:24.9437980Z a1d16b6a5070: Verifying Checksum
2023-01-11T21:05:24.9438782Z a1d16b6a5070: Download complete
2023-01-11T21:05:25.0314344Z bdf297d7f88c: Verifying Checksum
2023-01-11T21:05:25.0314687Z bdf297d7f88c: Download complete
2023-01-11T21:05:25.0967681Z 25571655d0e1: Download complete
2023-01-11T21:05:25.1815666Z 6d68f7da8baa: Download complete
2023-01-11T21:05:25.2637092Z cca768f96df4: Verifying Checksum
2023-01-11T21:05:25.2637422Z cca768f96df4: Download complete
2023-01-11T21:05:25.6438476Z 0b3950af8ae1: Verifying Checksum
2023-01-11T21:05:25.6439038Z 0b3950af8ae1: Download complete
2023-01-11T21:05:25.7289517Z 61eecfa8b34e: Download complete
2023-01-11T21:05:25.8084029Z 95c1ac011645: Verifying Checksum
2023-01-11T21:05:25.8084653Z 95c1ac011645: Download complete
2023-01-11T21:05:25.8760278Z 3046cc00c4ca: Download complete
2023-01-11T21:05:26.0534849Z 195d560d8cf6: Download complete
2023-01-11T21:05:26.2370184Z 77250abd5ca4: Verifying Checksum
2023-01-11T21:05:26.2370563Z 77250abd5ca4: Download complete
2023-01-11T21:05:26.3277112Z 881b24daf9c5: Verifying Checksum
2023-01-11T21:05:26.3277737Z 881b24daf9c5: Download complete
2023-01-11T21:05:27.8340724Z 9fbf0a18619e: Verifying Checksum
2023-01-11T21:05:27.8341349Z 9fbf0a18619e: Download complete
2023-01-11T21:05:27.9111884Z 02048a597c22: Verifying Checksum
2023-01-11T21:05:27.9112447Z 02048a597c22: Download complete
2023-01-11T21:05:31.3948124Z 8c3cf3d5e1c5: Verifying Checksum
2023-01-11T21:05:31.5004281Z 3e03143da3c2: Verifying Checksum
2023-01-11T21:05:31.5004818Z 3e03143da3c2: Download complete
2023-01-11T21:05:34.6696477Z 502253a1be21: Pull complete
2023-01-11T21:05:34.7709317Z 5c7dd67e5809: Pull complete
2023-01-11T21:05:34.9067211Z bdfd23ed3f48: Pull complete
2023-01-11T21:05:34.9958311Z aee1dd761bdd: Pull complete
2023-01-11T21:05:56.7228083Z 5feda9af2542: Pull complete
2023-01-11T21:05:58.6335612Z f8371ecb849a: Pull complete
2023-01-11T21:06:00.5728954Z ce4a87d45645: Pull complete
2023-01-11T21:06:08.5333444Z 39629f7269f9: Pull complete
2023-01-11T21:06:10.3818399Z 87d0ffa55850: Pull complete
2023-01-11T21:06:12.2585331Z 70702f8b5bc4: Pull complete
2023-01-11T21:06:12.6134608Z c7726d39d806: Verifying Checksum
2023-01-11T21:06:12.6135162Z c7726d39d806: Download complete
2023-01-11T21:06:14.1897247Z 0c06be5c20e0: Pull complete
2023-01-11T21:06:18.3843369Z b372c2a3bc3f: Pull complete
2023-01-11T21:06:20.6719066Z 582d081a59fa: Pull complete
2023-01-11T21:06:22.8773184Z e1c655e7ec0e: Pull complete
2023-01-11T21:06:55.4634679Z c7726d39d806: Pull complete
2023-01-11T21:06:57.3371871Z 1c22f2f8c01b: Pull complete
2023-01-11T21:06:59.1823521Z b8f759fd0191: Pull complete
2023-01-11T21:07:01.1533621Z e28e73a4bddd: Pull complete
2023-01-11T21:07:01.9503017Z 859052a25d95: Download complete
2023-01-11T21:07:03.0124561Z 90d8f9bbe048: Pull complete
2023-01-11T21:07:05.1379958Z b34bd39d0461: Pull complete
2023-01-11T21:07:06.9566911Z 2f2308643d60: Pull complete
2023-01-11T21:07:11.2957620Z 8e3432e5a569: Pull complete
2023-01-11T21:07:13.1718691Z 9ea746919509: Pull complete
2023-01-11T21:07:15.0366430Z 1a2fd7b216d7: Pull complete
2023-01-11T21:07:17.8283088Z 19fde6a723a0: Pull complete
2023-01-11T21:07:20.1615052Z 06369252d749: Pull complete
2023-01-11T21:07:23.7928852Z ea4bfeaa0fc7: Pull complete
2023-01-11T21:07:32.5212985Z a1d16b6a5070: Pull complete
2023-01-11T21:07:34.4168368Z f550b7ff2470: Pull complete
2023-01-11T21:07:36.0676651Z 12ddc57b99eb: Pull complete
2023-01-11T21:07:38.5015685Z 8345085fb0a0: Pull complete
2023-01-11T21:07:38.6043686Z 4cc94dbec031: Pull complete
2023-01-11T21:07:38.7153490Z 29a7c0d5fa4c: Pull complete
2023-01-11T21:07:39.0717443Z 25571655d0e1: Pull complete
2023-01-11T21:07:39.1943153Z bdf297d7f88c: Pull complete
2023-01-11T21:07:40.6226587Z 0b3950af8ae1: Pull complete
2023-01-11T21:07:40.7297268Z 6d68f7da8baa: Pull complete
2023-01-11T21:07:40.8424491Z cca768f96df4: Pull complete
2023-01-11T21:07:47.1741023Z 8c3cf3d5e1c5: Pull complete
2023-01-11T21:07:48.9912009Z 61eecfa8b34e: Pull complete
2023-01-11T21:07:50.8699263Z 95c1ac011645: Pull complete
2023-01-11T21:07:52.1106928Z 3046cc00c4ca: Pull complete
2023-01-11T21:07:52.2141441Z 195d560d8cf6: Pull complete
2023-01-11T21:07:52.9910937Z 77250abd5ca4: Pull complete
2023-01-11T21:07:53.0995573Z 881b24daf9c5: Pull complete
2023-01-11T21:07:54.9770380Z 9fbf0a18619e: Pull complete
2023-01-11T21:07:55.0971341Z 02048a597c22: Pull complete
2023-01-11T21:08:37.1281541Z 859052a25d95: Pull complete
2023-01-11T21:08:39.0038093Z 3e03143da3c2: Pull complete
2023-01-11T21:08:40.3202510Z Digest: sha256:866df6c1171dbe014496717cf2080d6cc72ca611a4e8146525c9ef09640c8ba4
2023-01-11T21:08:40.8205338Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:08:41.1034864Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:08:41.1138458Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2023-01-11T21:08:41.1138828Z with:
2023-01-11T21:08:41.1139071Z   driver-version: 515.76
2023-01-11T21:08:41.1139296Z env:
2023-01-11T21:08:41.1139535Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:08:41.1139793Z ##[endgroup]
2023-01-11T21:08:41.2608273Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T21:08:41.2608597Z with:
2023-01-11T21:08:41.2608814Z   timeout_minutes: 10
2023-01-11T21:08:41.2609070Z   max_attempts: 3
2023-01-11T21:08:41.2616187Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        sudo apt-get install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2023-01-11T21:08:41.2623157Z   retry_wait_seconds: 10
2023-01-11T21:08:41.2623448Z   polling_interval_seconds: 1
2023-01-11T21:08:41.2623725Z   warning_on_retry: true
2023-01-11T21:08:41.2623972Z   continue_on_error: false
2023-01-11T21:08:41.2624290Z env:
2023-01-11T21:08:41.2624533Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:08:41.2624783Z   DRIVER_VERSION: 515.76
2023-01-11T21:08:41.2625041Z ##[endgroup]
2023-01-11T21:08:41.3302282Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2023-01-11T21:08:41.3303985Z + pre_install_nvidia_driver_amzn2
2023-01-11T21:08:41.3304842Z + sudo yum remove -y nvidia-driver-latest-dkms
2023-01-11T21:08:41.8003953Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:08:41.8435632Z No Match for argument: nvidia-driver-latest-dkms
2023-01-11T21:08:41.8728178Z No Packages marked for removal
2023-01-11T21:08:41.8892076Z + install_nvidia_driver_common
2023-01-11T21:08:41.8894567Z + echo 'Before installing NVIDIA driver'
2023-01-11T21:08:41.8895629Z + lspci
2023-01-11T21:08:41.8897600Z Before installing NVIDIA driver
2023-01-11T21:08:43.1168456Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-01-11T21:08:43.1169170Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2023-01-11T21:08:43.1171043Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2023-01-11T21:08:43.1171522Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2023-01-11T21:08:43.1171979Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2023-01-11T21:08:43.1172402Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2023-01-11T21:08:43.1172869Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:08:43.1173322Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:08:43.1173738Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2023-01-11T21:08:43.1174087Z + lsmod
2023-01-11T21:08:43.1195495Z Module                  Size  Used by
2023-01-11T21:08:43.1195827Z xt_conntrack           16384  1
2023-01-11T21:08:43.1196151Z ipt_MASQUERADE         16384  1
2023-01-11T21:08:43.1196499Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2023-01-11T21:08:43.1198702Z nf_conntrack_netlink    49152  0
2023-01-11T21:08:43.1201447Z nfnetlink              16384  2 nf_conntrack_netlink
2023-01-11T21:08:43.1201802Z xfrm_user              45056  1
2023-01-11T21:08:43.1202120Z xfrm_algo              16384  1 xfrm_user
2023-01-11T21:08:43.1202416Z xt_addrtype            16384  2
2023-01-11T21:08:43.1202702Z iptable_filter         16384  1
2023-01-11T21:08:43.1202988Z iptable_nat            16384  1
2023-01-11T21:08:43.1203276Z nf_conntrack_ipv4      16384  3
2023-01-11T21:08:43.1203562Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2023-01-11T21:08:43.1203896Z nf_nat_ipv4            16384  1 iptable_nat
2023-01-11T21:08:43.1204276Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2023-01-11T21:08:43.1204748Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2023-01-11T21:08:43.1206211Z br_netfilter           24576  0
2023-01-11T21:08:43.1206541Z bridge                172032  1 br_netfilter
2023-01-11T21:08:43.1207243Z stp                    16384  1 bridge
2023-01-11T21:08:43.1207568Z llc                    16384  2 bridge,stp
2023-01-11T21:08:43.1207863Z overlay                86016  0
2023-01-11T21:08:43.1208154Z sunrpc                393216  1
2023-01-11T21:08:43.1208435Z dm_mirror              28672  0
2023-01-11T21:08:43.1208745Z dm_region_hash         20480  1 dm_mirror
2023-01-11T21:08:43.1209079Z dm_log                 20480  2 dm_region_hash,dm_mirror
2023-01-11T21:08:43.1209384Z dm_mod                143360  2 dm_log,dm_mirror
2023-01-11T21:08:43.1209686Z dax                    69632  1 dm_mod
2023-01-11T21:08:43.1209980Z sb_edac                24576  0
2023-01-11T21:08:43.1210241Z crc32_pclmul           16384  0
2023-01-11T21:08:43.1210550Z ghash_clmulni_intel    16384  0
2023-01-11T21:08:43.1210861Z pcbc                   16384  0
2023-01-11T21:08:43.1211412Z aesni_intel           188416  0
2023-01-11T21:08:43.1211702Z ata_piix               36864  0
2023-01-11T21:08:43.1212002Z aes_x86_64             20480  1 aesni_intel
2023-01-11T21:08:43.1212332Z crypto_simd            16384  1 aesni_intel
2023-01-11T21:08:43.1212617Z libata                266240  1 ata_piix
2023-01-11T21:08:43.1212931Z glue_helper            16384  1 aesni_intel
2023-01-11T21:08:43.1213312Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2023-01-11T21:08:43.1213638Z pcc_cpufreq            16384  0
2023-01-11T21:08:43.1213949Z mousedev               24576  0
2023-01-11T21:08:43.1214230Z evdev                  20480  3
2023-01-11T21:08:43.1214481Z scsi_mod              245760  1 libata
2023-01-11T21:08:43.1214773Z psmouse                32768  0
2023-01-11T21:08:43.1215053Z button                 16384  0
2023-01-11T21:08:43.1215421Z ena                   114688  0
2023-01-11T21:08:43.1215738Z xen_blkfront           49152  2
2023-01-11T21:08:43.1216032Z crc32c_intel           24576  0
2023-01-11T21:08:43.1216292Z autofs4                49152  2
2023-01-11T21:08:43.1216564Z + modinfo nvidia
2023-01-11T21:08:43.1216865Z modinfo: ERROR: Module nvidia not found.
2023-01-11T21:08:43.1217159Z + true
2023-01-11T21:08:43.1217383Z + HAS_NVIDIA_DRIVER=0
2023-01-11T21:08:43.1217815Z ++ command -v nvidia-smi
2023-01-11T21:08:43.1218133Z + '[' -x '' ']'
2023-01-11T21:08:43.1219384Z + '[' 0 -eq 0 ']'
2023-01-11T21:08:43.1219757Z + '[' amzn2 '!=' ubuntu20.04 ']'
2023-01-11T21:08:43.1220145Z + sudo yum groupinstall -y 'Development Tools'
2023-01-11T21:08:43.6002187Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:08:43.9385810Z Resolving Dependencies
2023-01-11T21:08:43.9391442Z --> Running transaction check
2023-01-11T21:08:43.9394663Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2023-01-11T21:08:43.9613167Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2023-01-11T21:08:44.1814108Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2023-01-11T21:08:44.1817714Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2023-01-11T21:08:44.1865794Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2023-01-11T21:08:44.1873520Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2023-01-11T21:08:44.1885221Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2023-01-11T21:08:44.2004887Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2023-01-11T21:08:44.2012848Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2023-01-11T21:08:44.2061139Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2023-01-11T21:08:44.2086427Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2023-01-11T21:08:44.2095852Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2023-01-11T21:08:44.2104483Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2023-01-11T21:08:44.2210736Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2023-01-11T21:08:44.2354757Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2023-01-11T21:08:44.2375589Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.2556712Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.2577463Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.2636082Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.2690907Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.2748044Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.2803736Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.2859881Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.2916497Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.3089298Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.3111988Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.3135083Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.3164097Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.3200524Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2023-01-11T21:08:44.3273466Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2023-01-11T21:08:44.3289249Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2023-01-11T21:08:44.3337384Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2023-01-11T21:08:44.3353687Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2023-01-11T21:08:44.3374058Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2023-01-11T21:08:44.3405236Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2023-01-11T21:08:44.3450880Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2023-01-11T21:08:44.3474975Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2023-01-11T21:08:44.3509742Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2023-01-11T21:08:44.3752749Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2023-01-11T21:08:44.3772007Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2023-01-11T21:08:44.3796729Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2023-01-11T21:08:44.3977951Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.3999209Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4000796Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4001701Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4002712Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4004118Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4005062Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4006164Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4006826Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4007915Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4008712Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4009656Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4010620Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4011389Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4012001Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4012854Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4031449Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4053698Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2023-01-11T21:08:44.4079545Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2023-01-11T21:08:44.4101924Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2023-01-11T21:08:44.4151252Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2023-01-11T21:08:44.4169991Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2023-01-11T21:08:44.4183267Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2023-01-11T21:08:44.4368844Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2023-01-11T21:08:44.4382861Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2023-01-11T21:08:44.4405933Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2023-01-11T21:08:44.4420555Z --> Running transaction check
2023-01-11T21:08:44.4424077Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2023-01-11T21:08:44.4503942Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2023-01-11T21:08:44.4544597Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2023-01-11T21:08:44.4560670Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.4636086Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2023-01-11T21:08:44.4648092Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2023-01-11T21:08:44.4649450Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2023-01-11T21:08:44.4724122Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2023-01-11T21:08:44.4785335Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2023-01-11T21:08:44.4794783Z ---> Package glibc-devel.x86_64 0:2.26-62.amzn2 will be installed
2023-01-11T21:08:44.4921523Z --> Processing Dependency: glibc-headers = 2.26-62.amzn2 for package: glibc-devel-2.26-62.amzn2.x86_64
2023-01-11T21:08:44.4950501Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-62.amzn2.x86_64
2023-01-11T21:08:44.4951482Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2 will be installed
2023-01-11T21:08:44.4956814Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.4971722Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.5000064Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.5038123Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.5055723Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2023-01-11T21:08:44.5070344Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.5086743Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.5113550Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2023-01-11T21:08:44.5162069Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2023-01-11T21:08:44.5178313Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2023-01-11T21:08:44.5201332Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2023-01-11T21:08:44.5280395Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2023-01-11T21:08:44.5321426Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2023-01-11T21:08:44.5322880Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2023-01-11T21:08:44.5343083Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2023-01-11T21:08:44.5362554Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2023-01-11T21:08:44.5369350Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2023-01-11T21:08:44.5419079Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2023-01-11T21:08:44.5519912Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2023-01-11T21:08:44.5533642Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2023-01-11T21:08:44.5549818Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2023-01-11T21:08:44.5551124Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2023-01-11T21:08:44.5581618Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2023-01-11T21:08:44.5803577Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2023-01-11T21:08:44.5818255Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2023-01-11T21:08:44.5845957Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2023-01-11T21:08:44.5846768Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2023-01-11T21:08:44.5971109Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2023-01-11T21:08:44.7080144Z --> Running transaction check
2023-01-11T21:08:44.7081718Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2023-01-11T21:08:44.7092040Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2023-01-11T21:08:44.7120330Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2023-01-11T21:08:44.7121890Z ---> Package glibc-headers.x86_64 0:2.26-62.amzn2 will be installed
2023-01-11T21:08:44.7200176Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-62.amzn2.x86_64
2023-01-11T21:08:44.8390813Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-62.amzn2.x86_64
2023-01-11T21:08:44.8391411Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2023-01-11T21:08:44.8461043Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2023-01-11T21:08:44.8488104Z ---> Package kernel-devel.x86_64 0:4.14.301-224.520.amzn2 will be installed
2023-01-11T21:08:44.8517163Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.301-224.520.amzn2.x86_64
2023-01-11T21:08:44.8538738Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2023-01-11T21:08:44.8568999Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2023-01-11T21:08:44.8588243Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2023-01-11T21:08:44.8638857Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2023-01-11T21:08:44.8661191Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2023-01-11T21:08:44.8662541Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2023-01-11T21:08:44.8663313Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2023-01-11T21:08:44.8677877Z --> Running transaction check
2023-01-11T21:08:44.8679014Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2023-01-11T21:08:44.8698021Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2023-01-11T21:08:44.8711140Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2023-01-11T21:08:44.8740317Z ---> Package kernel-headers.x86_64 0:4.14.301-224.520.amzn2 will be installed
2023-01-11T21:08:44.8741632Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2023-01-11T21:08:44.8770144Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2023-01-11T21:08:44.8830639Z --> Running transaction check
2023-01-11T21:08:44.8831314Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2 will be installed
2023-01-11T21:08:45.1462452Z --> Finished Dependency Resolution
2023-01-11T21:08:45.2249564Z 
2023-01-11T21:08:45.2249985Z Dependencies Resolved
2023-01-11T21:08:45.2373456Z 
2023-01-11T21:08:45.2373755Z ================================================================================
2023-01-11T21:08:45.2374429Z  Package                 Arch     Version                    Repository    Size
2023-01-11T21:08:45.2375230Z ================================================================================
2023-01-11T21:08:45.2375833Z Installing for group install "Development Tools":
2023-01-11T21:08:45.2376410Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2023-01-11T21:08:45.2376829Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2023-01-11T21:08:45.2377265Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2023-01-11T21:08:45.2377674Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2023-01-11T21:08:45.2378097Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2023-01-11T21:08:45.2378573Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2023-01-11T21:08:45.2379000Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2023-01-11T21:08:45.2379408Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2023-01-11T21:08:45.2379837Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2023-01-11T21:08:45.2380283Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2023-01-11T21:08:45.2380674Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2023-01-11T21:08:45.2381073Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2023-01-11T21:08:45.2381495Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2023-01-11T21:08:45.2381929Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2023-01-11T21:08:45.2382333Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2023-01-11T21:08:45.2382766Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2023-01-11T21:08:45.2383184Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2023-01-11T21:08:45.2383616Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2023-01-11T21:08:45.2384032Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2023-01-11T21:08:45.2384450Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2023-01-11T21:08:45.2384883Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2023-01-11T21:08:45.2385296Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2023-01-11T21:08:45.2385718Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2023-01-11T21:08:45.2386155Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2023-01-11T21:08:45.2386602Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2023-01-11T21:08:45.2386897Z Installing for dependencies:
2023-01-11T21:08:45.2387303Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2023-01-11T21:08:45.2387729Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2023-01-11T21:08:45.2388306Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2023-01-11T21:08:45.2388742Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2023-01-11T21:08:45.2389169Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2023-01-11T21:08:45.2389587Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2023-01-11T21:08:45.2389995Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2023-01-11T21:08:45.2390448Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2023-01-11T21:08:45.2390908Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2023-01-11T21:08:45.2391333Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2023-01-11T21:08:45.2391842Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2023-01-11T21:08:45.2392318Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2023-01-11T21:08:45.2392761Z  glibc-devel             x86_64   2.26-62.amzn2              amzn2-core   995 k
2023-01-11T21:08:45.2393179Z  glibc-headers           x86_64   2.26-62.amzn2              amzn2-core   516 k
2023-01-11T21:08:45.2393606Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2023-01-11T21:08:45.2394042Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.2        amzn2-core    23 k
2023-01-11T21:08:45.2394472Z  kernel-devel            x86_64   4.14.301-224.520.amzn2     amzn2-core    13 M
2023-01-11T21:08:45.2395172Z  kernel-headers          x86_64   4.14.301-224.520.amzn2     amzn2-core   1.2 M
2023-01-11T21:08:45.2395956Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2023-01-11T21:08:45.2396614Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2023-01-11T21:08:45.2397032Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2023-01-11T21:08:45.2397443Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2023-01-11T21:08:45.2397869Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2023-01-11T21:08:45.2398282Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2023-01-11T21:08:45.2398704Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2023-01-11T21:08:45.2399129Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2023-01-11T21:08:45.2399935Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2023-01-11T21:08:45.2400348Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2023-01-11T21:08:45.2400770Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2023-01-11T21:08:45.2401193Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2023-01-11T21:08:45.2401603Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2023-01-11T21:08:45.2402018Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2023-01-11T21:08:45.2402446Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2023-01-11T21:08:45.2402893Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2023-01-11T21:08:45.2403342Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2023-01-11T21:08:45.2403801Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2023-01-11T21:08:45.2404268Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2023-01-11T21:08:45.2404715Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2023-01-11T21:08:45.2405177Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2023-01-11T21:08:45.2405625Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2023-01-11T21:08:45.2406204Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2023-01-11T21:08:45.2406622Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2023-01-11T21:08:45.2407049Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.2         amzn2-core    50 k
2023-01-11T21:08:45.2407256Z 
2023-01-11T21:08:45.2407370Z Transaction Summary
2023-01-11T21:08:45.2407640Z ================================================================================
2023-01-11T21:08:45.2407953Z Install  25 Packages (+43 Dependent packages)
2023-01-11T21:08:45.2408153Z 
2023-01-11T21:08:45.2408288Z Total download size: 96 M
2023-01-11T21:08:45.2408551Z Installed size: 303 M
2023-01-11T21:08:45.2408793Z Downloading packages:
2023-01-11T21:08:45.2431329Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2023-01-11T21:08:46.4949656Z --------------------------------------------------------------------------------
2023-01-11T21:08:46.4950137Z Total                                               77 MB/s |  96 MB  00:01     
2023-01-11T21:08:46.6017691Z Running transaction check
2023-01-11T21:08:46.6792860Z Running transaction test
2023-01-11T21:08:48.7128493Z Transaction test succeeded
2023-01-11T21:08:48.7131632Z Running transaction
2023-01-11T21:08:53.7823911Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2023-01-11T21:08:56.2533310Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2023-01-11T21:08:58.8025319Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2023-01-11T21:09:01.2346772Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2023-01-11T21:09:03.6878484Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2023-01-11T21:09:06.1269353Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2023-01-11T21:09:08.5798307Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2023-01-11T21:09:11.0185428Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2023-01-11T21:09:11.1730815Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2023-01-11T21:09:11.2560588Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2023-01-11T21:09:12.3190236Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2023-01-11T21:09:12.3614047Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2023-01-11T21:09:12.3992243Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         13/68 
2023-01-11T21:09:12.4633689Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        14/68 
2023-01-11T21:09:12.5229635Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   15/68 
2023-01-11T21:09:12.6331433Z   Installing : elfutils-0.176-2.amzn2.x86_64                              16/68 
2023-01-11T21:09:12.6651757Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         17/68 
2023-01-11T21:09:12.6910071Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 18/68 
2023-01-11T21:09:12.7247118Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          19/68 
2023-01-11T21:09:12.7538957Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        20/68 
2023-01-11T21:09:12.8336087Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               21/68 
2023-01-11T21:09:12.8875394Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      22/68 
2023-01-11T21:09:12.9859944Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2023-01-11T21:09:13.1664139Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2023-01-11T21:09:13.2062900Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2023-01-11T21:09:13.6007550Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               26/68 
2023-01-11T21:09:13.6292901Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              27/68 
2023-01-11T21:09:13.6607668Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               28/68 
2023-01-11T21:09:13.6811335Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      29/68 
2023-01-11T21:09:13.7127206Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  30/68 
2023-01-11T21:09:13.7414149Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               31/68 
2023-01-11T21:09:13.8355859Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  32/68 
2023-01-11T21:09:13.9258769Z   Installing : autoconf-2.69-11.amzn2.noarch                              33/68 
2023-01-11T21:09:14.0450035Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      34/68 
2023-01-11T21:09:14.0872754Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           35/68 
2023-01-11T21:09:14.1109596Z   Installing : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       36/68 
2023-01-11T21:09:14.1351869Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 37/68 
2023-01-11T21:09:14.4317656Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            38/68 
2023-01-11T21:09:14.6071245Z   Installing : kernel-headers-4.14.301-224.520.amzn2.x86_64               39/68 
2023-01-11T21:09:14.7433612Z   Installing : glibc-headers-2.26-62.amzn2.x86_64                         40/68 
2023-01-11T21:09:14.7903371Z   Installing : glibc-devel-2.26-62.amzn2.x86_64                           41/68 
2023-01-11T21:09:16.8361526Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2023-01-11T21:09:20.8164025Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2023-01-11T21:09:32.0665063Z   Installing : kernel-devel-4.14.301-224.520.amzn2.x86_64                 44/68 
2023-01-11T21:09:32.7121575Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2023-01-11T21:09:32.7848063Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2023-01-11T21:09:32.8445627Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2023-01-11T21:09:32.9845372Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2023-01-11T21:09:33.1674310Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2023-01-11T21:09:33.2778854Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2023-01-11T21:09:34.5005493Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2023-01-11T21:09:36.1459068Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2023-01-11T21:09:36.2627043Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2023-01-11T21:09:36.3012735Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2023-01-11T21:09:36.3449065Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2023-01-11T21:09:36.4021954Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2023-01-11T21:09:36.4648742Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2023-01-11T21:09:36.5735035Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2023-01-11T21:09:36.6458971Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2023-01-11T21:09:36.6939617Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2023-01-11T21:09:36.7349171Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2023-01-11T21:09:36.7802536Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2023-01-11T21:09:37.4868023Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2023-01-11T21:09:37.5711552Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2023-01-11T21:09:37.6178373Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2023-01-11T21:09:37.9585388Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2023-01-11T21:09:38.0053677Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2023-01-11T21:09:38.0750926Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2023-01-11T21:09:38.0880750Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2023-01-11T21:09:38.0993948Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2023-01-11T21:09:38.1114545Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2023-01-11T21:09:38.1254558Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2023-01-11T21:09:38.1378557Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              5/68 
2023-01-11T21:09:38.1527948Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            6/68 
2023-01-11T21:09:38.1654331Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         7/68 
2023-01-11T21:09:38.1760718Z   Verifying  : glibc-devel-2.26-62.amzn2.x86_64                            8/68 
2023-01-11T21:09:38.1876996Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                          9/68 
2023-01-11T21:09:38.1993995Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            10/68 
2023-01-11T21:09:38.2123789Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      11/68 
2023-01-11T21:09:38.2238564Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            12/68 
2023-01-11T21:09:38.2357843Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           13/68 
2023-01-11T21:09:38.2484215Z   Verifying  : kernel-headers-4.14.301-224.520.amzn2.x86_64               14/68 
2023-01-11T21:09:38.2601258Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        15/68 
2023-01-11T21:09:38.2708406Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            16/68 
2023-01-11T21:09:38.2829444Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           17/68 
2023-01-11T21:09:38.2936707Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             18/68 
2023-01-11T21:09:38.3036444Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              19/68 
2023-01-11T21:09:38.3123944Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            20/68 
2023-01-11T21:09:38.3225671Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               21/68 
2023-01-11T21:09:38.3321430Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     22/68 
2023-01-11T21:09:38.3427674Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       23/68 
2023-01-11T21:09:38.3531441Z   Verifying  : glibc-headers-2.26-62.amzn2.x86_64                         24/68 
2023-01-11T21:09:38.3635982Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      25/68 
2023-01-11T21:09:38.3742362Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              26/68 
2023-01-11T21:09:38.3861232Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          27/68 
2023-01-11T21:09:38.3989463Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             28/68 
2023-01-11T21:09:38.4099855Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          29/68 
2023-01-11T21:09:38.4220202Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  30/68 
2023-01-11T21:09:38.4332887Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       31/68 
2023-01-11T21:09:38.4433288Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  32/68 
2023-01-11T21:09:38.4530985Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  33/68 
2023-01-11T21:09:38.4635209Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      34/68 
2023-01-11T21:09:38.4754898Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           35/68 
2023-01-11T21:09:38.4882976Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 36/68 
2023-01-11T21:09:38.5001719Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           37/68 
2023-01-11T21:09:38.5141593Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      38/68 
2023-01-11T21:09:38.5260237Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               39/68 
2023-01-11T21:09:38.5461879Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          40/68 
2023-01-11T21:09:38.5579897Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             41/68 
2023-01-11T21:09:38.5685287Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          42/68 
2023-01-11T21:09:38.5814115Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              43/68 
2023-01-11T21:09:38.5920891Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        44/68 
2023-01-11T21:09:38.6041413Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               45/68 
2023-01-11T21:09:38.6156543Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        46/68 
2023-01-11T21:09:38.6259438Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  47/68 
2023-01-11T21:09:38.6363905Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          48/68 
2023-01-11T21:09:38.6468988Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      49/68 
2023-01-11T21:09:38.6578091Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   50/68 
2023-01-11T21:09:38.6685360Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              51/68 
2023-01-11T21:09:38.6797198Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        52/68 
2023-01-11T21:09:38.6922024Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              53/68 
2023-01-11T21:09:38.7040028Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         54/68 
2023-01-11T21:09:38.7168421Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             55/68 
2023-01-11T21:09:38.7276721Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           56/68 
2023-01-11T21:09:38.7379456Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                57/68 
2023-01-11T21:09:38.7480279Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               58/68 
2023-01-11T21:09:38.7589535Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    59/68 
2023-01-11T21:09:38.7702481Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          60/68 
2023-01-11T21:09:38.7799722Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               61/68 
2023-01-11T21:09:38.7964627Z   Verifying  : kernel-devel-4.14.301-224.520.amzn2.x86_64                 62/68 
2023-01-11T21:09:38.8077875Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 63/68 
2023-01-11T21:09:38.8200647Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         64/68 
2023-01-11T21:09:38.8330554Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              65/68 
2023-01-11T21:09:38.8438573Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              66/68 
2023-01-11T21:09:38.8541253Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   67/68 
2023-01-11T21:09:38.9333372Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         68/68 
2023-01-11T21:09:38.9333627Z 
2023-01-11T21:09:38.9333744Z Installed:
2023-01-11T21:09:38.9334309Z   autoconf.noarch 0:2.69-11.amzn2                                               
2023-01-11T21:09:38.9335307Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2023-01-11T21:09:38.9335737Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2023-01-11T21:09:38.9336171Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2023-01-11T21:09:38.9342479Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2023-01-11T21:09:38.9343207Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2023-01-11T21:09:38.9343629Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2023-01-11T21:09:38.9344057Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2023-01-11T21:09:38.9344617Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2023-01-11T21:09:38.9345067Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2023-01-11T21:09:38.9345480Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2023-01-11T21:09:38.9345892Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2023-01-11T21:09:38.9346312Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2023-01-11T21:09:38.9346727Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2023-01-11T21:09:38.9347134Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2023-01-11T21:09:38.9347551Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2023-01-11T21:09:38.9347940Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2023-01-11T21:09:38.9348407Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2023-01-11T21:09:38.9348821Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2023-01-11T21:09:38.9349217Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2023-01-11T21:09:38.9349618Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2023-01-11T21:09:38.9350034Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2023-01-11T21:09:38.9350440Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2023-01-11T21:09:38.9350856Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2023-01-11T21:09:38.9351293Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2023-01-11T21:09:38.9351494Z 
2023-01-11T21:09:38.9351615Z Dependency Installed:
2023-01-11T21:09:38.9352001Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2023-01-11T21:09:38.9352398Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2023-01-11T21:09:38.9352823Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2023-01-11T21:09:38.9353245Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2023-01-11T21:09:38.9353643Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2023-01-11T21:09:38.9354096Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2023-01-11T21:09:38.9354513Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2023-01-11T21:09:38.9354955Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2023-01-11T21:09:38.9355390Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2023-01-11T21:09:38.9355817Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2023-01-11T21:09:38.9356254Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2023-01-11T21:09:38.9356688Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2023-01-11T21:09:38.9357188Z   glibc-devel.x86_64 0:2.26-62.amzn2                                            
2023-01-11T21:09:38.9357609Z   glibc-headers.x86_64 0:2.26-62.amzn2                                          
2023-01-11T21:09:38.9358030Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2023-01-11T21:09:38.9358449Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2                                   
2023-01-11T21:09:38.9358889Z   kernel-devel.x86_64 0:4.14.301-224.520.amzn2                                  
2023-01-11T21:09:38.9359318Z   kernel-headers.x86_64 0:4.14.301-224.520.amzn2                                
2023-01-11T21:09:38.9360033Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2023-01-11T21:09:38.9360536Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2023-01-11T21:09:38.9360964Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2023-01-11T21:09:38.9361381Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2023-01-11T21:09:38.9361767Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2023-01-11T21:09:38.9362178Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2023-01-11T21:09:38.9362579Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2023-01-11T21:09:38.9362966Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2023-01-11T21:09:38.9363373Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2023-01-11T21:09:38.9363789Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2023-01-11T21:09:38.9364191Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2023-01-11T21:09:38.9364576Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2023-01-11T21:09:38.9364980Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2023-01-11T21:09:38.9365375Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2023-01-11T21:09:38.9365767Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2023-01-11T21:09:38.9366199Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2023-01-11T21:09:38.9366648Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2023-01-11T21:09:38.9367091Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2023-01-11T21:09:38.9367552Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2023-01-11T21:09:38.9368004Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2023-01-11T21:09:38.9368456Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2023-01-11T21:09:38.9368874Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2023-01-11T21:09:38.9369320Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2023-01-11T21:09:38.9369744Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2023-01-11T21:09:38.9370135Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2                                        
2023-01-11T21:09:38.9370335Z 
2023-01-11T21:09:38.9370438Z Complete!
2023-01-11T21:09:38.9727667Z ++ uname -r
2023-01-11T21:09:38.9736264Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2023-01-11T21:09:39.4814758Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:09:39.7698461Z Resolving Dependencies
2023-01-11T21:09:39.7704841Z --> Running transaction check
2023-01-11T21:09:39.7705466Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2023-01-11T21:09:40.0601353Z --> Finished Dependency Resolution
2023-01-11T21:09:40.1405745Z 
2023-01-11T21:09:40.1406426Z Dependencies Resolved
2023-01-11T21:09:40.1411769Z 
2023-01-11T21:09:40.1412248Z ================================================================================
2023-01-11T21:09:40.1412973Z  Package          Arch       Version                       Repository      Size
2023-01-11T21:09:40.1413712Z ================================================================================
2023-01-11T21:09:40.1414030Z Installing:
2023-01-11T21:09:40.1414859Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2023-01-11T21:09:40.1415325Z 
2023-01-11T21:09:40.1415551Z Transaction Summary
2023-01-11T21:09:40.1416127Z ================================================================================
2023-01-11T21:09:40.1416667Z Install  1 Package
2023-01-11T21:09:40.1416836Z 
2023-01-11T21:09:40.1416974Z Total download size: 13 M
2023-01-11T21:09:40.1417377Z Installed size: 60 M
2023-01-11T21:09:40.1417637Z Downloading packages:
2023-01-11T21:09:40.1425279Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2023-01-11T21:09:40.4356463Z Running transaction check
2023-01-11T21:09:40.4548816Z Running transaction test
2023-01-11T21:09:40.8617786Z Transaction test succeeded
2023-01-11T21:09:40.8620602Z Running transaction
2023-01-11T21:09:56.0382114Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2023-01-11T21:09:56.1192902Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2023-01-11T21:09:56.1193203Z 
2023-01-11T21:09:56.1193294Z Installed:
2023-01-11T21:09:56.1193699Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2023-01-11T21:09:56.1193908Z 
2023-01-11T21:09:56.1194019Z Complete!
2023-01-11T21:09:56.1520388Z + sudo modprobe backlight
2023-01-11T21:09:56.1746730Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.76.run
2023-01-11T21:09:59.8537005Z + set +e
2023-01-11T21:09:59.8537864Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2023-01-11T21:10:01.2311982Z Verifying archive integrity... OK
2023-01-11T21:10:28.1494918Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.76................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2023-01-11T21:10:28.2918774Z 
2023-01-11T21:10:28.2919720Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2023-01-11T21:10:28.2920078Z 
2023-01-11T21:10:42.5518586Z 
2023-01-11T21:10:42.5520204Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2023-01-11T21:10:42.5521820Z 
2023-01-11T21:10:50.9929884Z + NVIDIA_INSTALLATION_STATUS=0
2023-01-11T21:10:50.9930253Z + RESET_GPU=0
2023-01-11T21:10:50.9930636Z + '[' 0 -ne 0 ']'
2023-01-11T21:10:50.9932873Z ++ command -v nvidia-smi
2023-01-11T21:10:50.9935553Z + '[' -x /usr/bin/nvidia-smi ']'
2023-01-11T21:10:50.9939198Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2023-01-11T21:10:56.0730850Z + INSTALLED_DRIVER_VERSION=515.76
2023-01-11T21:10:56.0731183Z + NVIDIA_SMI_STATUS=0
2023-01-11T21:10:56.0731599Z + '[' 0 -ne 0 ']'
2023-01-11T21:10:56.0731839Z + '[' 0 -eq 1 ']'
2023-01-11T21:10:56.0735179Z + sudo rm -fv /tmp/nvidia_driver
2023-01-11T21:10:56.1343082Z removed ‘/tmp/nvidia_driver’
2023-01-11T21:10:56.1357791Z + set -e
2023-01-11T21:10:56.1359983Z + post_install_nvidia_driver_common
2023-01-11T21:10:56.1363232Z + sudo modprobe nvidia
2023-01-11T21:10:56.1480752Z + echo 'After installing NVIDIA driver'
2023-01-11T21:10:56.1481024Z + lspci
2023-01-11T21:10:56.1481256Z After installing NVIDIA driver
2023-01-11T21:10:56.1686745Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-01-11T21:10:56.1687223Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2023-01-11T21:10:56.1687643Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2023-01-11T21:10:56.1688008Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2023-01-11T21:10:56.1688371Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2023-01-11T21:10:56.1688755Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2023-01-11T21:10:56.1689188Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:10:56.1690471Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:10:56.1690900Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2023-01-11T21:10:56.1691211Z + lsmod
2023-01-11T21:10:56.1708010Z Module                  Size  Used by
2023-01-11T21:10:56.1708287Z nvidia              40808448  0
2023-01-11T21:10:56.1708628Z drm                   425984  1 nvidia
2023-01-11T21:10:56.1708930Z i2c_core               77824  2 nvidia,drm
2023-01-11T21:10:56.1710671Z backlight              16384  0
2023-01-11T21:10:56.1711420Z xt_conntrack           16384  1
2023-01-11T21:10:56.1711726Z ipt_MASQUERADE         16384  1
2023-01-11T21:10:56.1712333Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2023-01-11T21:10:56.1712780Z nf_conntrack_netlink    49152  0
2023-01-11T21:10:56.1713377Z nfnetlink              16384  2 nf_conntrack_netlink
2023-01-11T21:10:56.1713710Z xfrm_user              45056  1
2023-01-11T21:10:56.1714041Z xfrm_algo              16384  1 xfrm_user
2023-01-11T21:10:56.1714614Z xt_addrtype            16384  2
2023-01-11T21:10:56.1715047Z iptable_filter         16384  1
2023-01-11T21:10:56.1715370Z iptable_nat            16384  1
2023-01-11T21:10:56.1716381Z nf_conntrack_ipv4      16384  3
2023-01-11T21:10:56.1716861Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2023-01-11T21:10:56.1717251Z nf_nat_ipv4            16384  1 iptable_nat
2023-01-11T21:10:56.1718084Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2023-01-11T21:10:56.1718987Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2023-01-11T21:10:56.1719474Z br_netfilter           24576  0
2023-01-11T21:10:56.1720043Z bridge                172032  1 br_netfilter
2023-01-11T21:10:56.1720439Z stp                    16384  1 bridge
2023-01-11T21:10:56.1720701Z llc                    16384  2 bridge,stp
2023-01-11T21:10:56.1721079Z overlay                86016  0
2023-01-11T21:10:56.1721405Z sunrpc                393216  1
2023-01-11T21:10:56.1721678Z dm_mirror              28672  0
2023-01-11T21:10:56.1721994Z dm_region_hash         20480  1 dm_mirror
2023-01-11T21:10:56.1722403Z dm_log                 20480  2 dm_region_hash,dm_mirror
2023-01-11T21:10:56.1722733Z dm_mod                143360  2 dm_log,dm_mirror
2023-01-11T21:10:56.1723064Z dax                    69632  1 dm_mod
2023-01-11T21:10:56.1723386Z sb_edac                24576  0
2023-01-11T21:10:56.1723823Z crc32_pclmul           16384  0
2023-01-11T21:10:56.1724153Z ghash_clmulni_intel    16384  0
2023-01-11T21:10:56.1724475Z pcbc                   16384  0
2023-01-11T21:10:56.1724733Z aesni_intel           188416  0
2023-01-11T21:10:56.1725034Z ata_piix               36864  0
2023-01-11T21:10:56.1725451Z aes_x86_64             20480  1 aesni_intel
2023-01-11T21:10:56.1725782Z crypto_simd            16384  1 aesni_intel
2023-01-11T21:10:56.1726115Z libata                266240  1 ata_piix
2023-01-11T21:10:56.1726430Z glue_helper            16384  1 aesni_intel
2023-01-11T21:10:56.1726887Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2023-01-11T21:10:56.1727274Z pcc_cpufreq            16384  0
2023-01-11T21:10:56.1727542Z mousedev               24576  0
2023-01-11T21:10:56.1727838Z evdev                  20480  3
2023-01-11T21:10:56.1728255Z scsi_mod              245760  1 libata
2023-01-11T21:10:56.1728582Z psmouse                32768  0
2023-01-11T21:10:56.1728884Z button                 16384  0
2023-01-11T21:10:56.1729186Z ena                   114688  0
2023-01-11T21:10:56.1729465Z xen_blkfront           49152  2
2023-01-11T21:10:56.1729770Z crc32c_intel           24576  0
2023-01-11T21:10:56.1730104Z autofs4                49152  2
2023-01-11T21:10:56.1730357Z + modinfo nvidia
2023-01-11T21:10:56.1730893Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2023-01-11T21:10:56.1731289Z firmware:       nvidia/515.76/gsp.bin
2023-01-11T21:10:56.1731634Z alias:          char-major-195-*
2023-01-11T21:10:56.1731991Z version:        515.76
2023-01-11T21:10:56.1732308Z supported:      external
2023-01-11T21:10:56.1732569Z license:        NVIDIA
2023-01-11T21:10:56.1732903Z srcversion:     51FD9DD90150B35351AFFBB
2023-01-11T21:10:56.1733292Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-01-11T21:10:56.1733670Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-01-11T21:10:56.1733983Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-01-11T21:10:56.1734372Z depends:        i2c-core,drm
2023-01-11T21:10:56.1734680Z retpoline:      Y
2023-01-11T21:10:56.1734969Z name:           nvidia
2023-01-11T21:10:56.1735429Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2023-01-11T21:10:56.1735881Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-01-11T21:10:56.1736298Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-01-11T21:10:56.1736732Z parm:           NVreg_ResmanDebugLevel:int
2023-01-11T21:10:56.1737149Z parm:           NVreg_RmLogonRC:int
2023-01-11T21:10:56.1737472Z parm:           NVreg_ModifyDeviceFiles:int
2023-01-11T21:10:56.1737847Z parm:           NVreg_DeviceFileUID:int
2023-01-11T21:10:56.1738209Z parm:           NVreg_DeviceFileGID:int
2023-01-11T21:10:56.1770827Z parm:           NVreg_DeviceFileMode:int
2023-01-11T21:10:56.1771403Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-01-11T21:10:56.1771848Z parm:           NVreg_UsePageAttributeTable:int
2023-01-11T21:10:56.1772169Z parm:           NVreg_EnablePCIeGen3:int
2023-01-11T21:10:56.1772445Z parm:           NVreg_EnableMSI:int
2023-01-11T21:10:56.1772739Z parm:           NVreg_TCEBypassMode:int
2023-01-11T21:10:56.1773067Z parm:           NVreg_EnableStreamMemOPs:int
2023-01-11T21:10:56.1773421Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-01-11T21:10:56.1773822Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-01-11T21:10:56.1774216Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-01-11T21:10:56.1774619Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-01-11T21:10:56.1775011Z parm:           NVreg_DynamicPowerManagement:int
2023-01-11T21:10:56.1775434Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-01-11T21:10:56.1775844Z parm:           NVreg_EnableGpuFirmware:int
2023-01-11T21:10:56.1776182Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-01-11T21:10:56.1776555Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-01-11T21:10:56.1777092Z parm:           NVreg_EnableUserNUMAManagement:int
2023-01-11T21:10:56.1777414Z parm:           NVreg_MemoryPoolSize:int
2023-01-11T21:10:56.1777749Z parm:           NVreg_KMallocHeapMaxSize:int
2023-01-11T21:10:56.1778092Z parm:           NVreg_VMallocHeapMaxSize:int
2023-01-11T21:10:56.1778397Z parm:           NVreg_IgnoreMMIOCheck:int
2023-01-11T21:10:56.1778710Z parm:           NVreg_NvLinkDisable:int
2023-01-11T21:10:56.1779141Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-01-11T21:10:56.1779512Z parm:           NVreg_RegisterPCIDriver:int
2023-01-11T21:10:56.1779832Z parm:           NVreg_EnableDbgBreakpoint:int
2023-01-11T21:10:56.1780164Z parm:           NVreg_RegistryDwords:charp
2023-01-11T21:10:56.1780514Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-01-11T21:10:56.1780821Z parm:           NVreg_RmMsg:charp
2023-01-11T21:10:56.1781193Z parm:           NVreg_GpuBlacklist:charp
2023-01-11T21:10:56.1781537Z parm:           NVreg_TemporaryFilePath:charp
2023-01-11T21:10:56.1781852Z parm:           NVreg_ExcludedGpus:charp
2023-01-11T21:10:56.1782177Z parm:           NVreg_DmaRemapPeerMmio:int
2023-01-11T21:10:56.1782498Z parm:           rm_firmware_active:charp
2023-01-11T21:10:56.1782787Z + set +e
2023-01-11T21:10:56.1783100Z + nvidia-smi
2023-01-11T21:10:59.9830259Z Wed Jan 11 21:10:59 2023       
2023-01-11T21:10:59.9831318Z +-----------------------------------------------------------------------------+
2023-01-11T21:10:59.9832075Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2023-01-11T21:10:59.9832538Z |-------------------------------+----------------------+----------------------+
2023-01-11T21:10:59.9833032Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2023-01-11T21:10:59.9833524Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2023-01-11T21:10:59.9833867Z |                               |                      |               MIG M. |
2023-01-11T21:10:59.9834151Z |===============================+======================+======================|
2023-01-11T21:10:59.9877655Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |                    0 |
2023-01-11T21:10:59.9878368Z | N/A   25C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2023-01-11T21:10:59.9879005Z |                               |                      |                  N/A |
2023-01-11T21:10:59.9879773Z +-------------------------------+----------------------+----------------------+
2023-01-11T21:10:59.9924941Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |                    0 |
2023-01-11T21:10:59.9925638Z | N/A   28C    P0    38W / 150W |      0MiB /  7680MiB |     99%      Default |
2023-01-11T21:10:59.9926289Z |                               |                      |                  N/A |
2023-01-11T21:10:59.9926833Z +-------------------------------+----------------------+----------------------+
2023-01-11T21:10:59.9927183Z                                                                                
2023-01-11T21:10:59.9927598Z +-----------------------------------------------------------------------------+
2023-01-11T21:10:59.9927953Z | Processes:                                                                  |
2023-01-11T21:10:59.9928279Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2023-01-11T21:10:59.9928592Z |        ID   ID                                                   Usage      |
2023-01-11T21:10:59.9928873Z |=============================================================================|
2023-01-11T21:10:59.9930252Z |  No running processes found                                                 |
2023-01-11T21:10:59.9931193Z +-----------------------------------------------------------------------------+
2023-01-11T21:11:00.5285465Z + NVIDIA_SMI_STATUS=0
2023-01-11T21:11:00.5286210Z + '[' 0 -eq 0 ']'
2023-01-11T21:11:00.5286829Z + echo 'INFO: Ignoring allowed status 0'
2023-01-11T21:11:00.5287312Z + set -e
2023-01-11T21:11:00.5287560Z INFO: Ignoring allowed status 0
2023-01-11T21:11:00.5292832Z == Installing nvidia container toolkit for amzn2 ==
2023-01-11T21:11:00.5296555Z + sudo yum install -y yum-utils
2023-01-11T21:11:01.0726507Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:11:01.3540776Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2023-01-11T21:11:01.3541266Z Nothing to do
2023-01-11T21:11:01.3753202Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2023-01-11T21:11:01.9032854Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:11:01.9307648Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2023-01-11T21:11:01.9309047Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2023-01-11T21:11:01.9309625Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2023-01-11T21:11:01.9454033Z + sudo yum install -y nvidia-docker2
2023-01-11T21:11:02.4684698Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:11:02.5082998Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2023-01-11T21:11:02.5162465Z Importing GPG key 0xF796ECB0:
2023-01-11T21:11:02.5163182Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2023-01-11T21:11:02.5164054Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2023-01-11T21:11:02.5164704Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2023-01-11T21:11:02.9065674Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2023-01-11T21:11:02.9190651Z Importing GPG key 0xF796ECB0:
2023-01-11T21:11:02.9191482Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2023-01-11T21:11:02.9192149Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2023-01-11T21:11:02.9192661Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2023-01-11T21:11:03.1423794Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2023-01-11T21:11:03.1506885Z Importing GPG key 0xF796ECB0:
2023-01-11T21:11:03.1507667Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2023-01-11T21:11:03.1508426Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2023-01-11T21:11:03.1508899Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2023-01-11T21:11:04.9360212Z Resolving Dependencies
2023-01-11T21:11:04.9366696Z --> Running transaction check
2023-01-11T21:11:04.9367609Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2023-01-11T21:11:04.9395131Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2023-01-11T21:11:05.0322605Z --> Running transaction check
2023-01-11T21:11:05.0323597Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed
2023-01-11T21:11:05.0468556Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2023-01-11T21:11:05.0481003Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2023-01-11T21:11:05.0609155Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2023-01-11T21:11:05.0610181Z --> Running transaction check
2023-01-11T21:11:05.0611091Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed
2023-01-11T21:11:05.0623119Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64
2023-01-11T21:11:05.0652689Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2023-01-11T21:11:05.0654194Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2023-01-11T21:11:05.0655822Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed
2023-01-11T21:11:05.0659302Z --> Running transaction check
2023-01-11T21:11:05.0660193Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed
2023-01-11T21:11:05.3056198Z --> Finished Dependency Resolution
2023-01-11T21:11:05.3786146Z 
2023-01-11T21:11:05.3786669Z Dependencies Resolved
2023-01-11T21:11:05.3800794Z 
2023-01-11T21:11:05.3801294Z ================================================================================
2023-01-11T21:11:05.3801715Z  Package                        Arch    Version      Repository            Size
2023-01-11T21:11:05.3802097Z ================================================================================
2023-01-11T21:11:05.3802350Z Installing:
2023-01-11T21:11:05.3803044Z  nvidia-docker2                 noarch  2.11.0-1     libnvidia-container  8.7 k
2023-01-11T21:11:05.3803421Z Installing for dependencies:
2023-01-11T21:11:05.3803884Z  libnvidia-container-tools      x86_64  1.11.0-1     libnvidia-container   49 k
2023-01-11T21:11:05.3804394Z  libnvidia-container1           x86_64  1.11.0-1     libnvidia-container  1.0 M
2023-01-11T21:11:05.3804906Z  nvidia-container-toolkit       x86_64  1.11.0-1     libnvidia-container  780 k
2023-01-11T21:11:05.3805461Z  nvidia-container-toolkit-base  x86_64  1.11.0-1     libnvidia-container  2.5 M
2023-01-11T21:11:05.3805690Z 
2023-01-11T21:11:05.3805803Z Transaction Summary
2023-01-11T21:11:05.3806098Z ================================================================================
2023-01-11T21:11:05.3806412Z Install  1 Package (+4 Dependent packages)
2023-01-11T21:11:05.3806606Z 
2023-01-11T21:11:05.3806719Z Total download size: 4.3 M
2023-01-11T21:11:05.3806980Z Installed size: 12 M
2023-01-11T21:11:05.3807629Z Downloading packages:
2023-01-11T21:11:05.4905916Z --------------------------------------------------------------------------------
2023-01-11T21:11:05.4906764Z Total                                               39 MB/s | 4.3 MB  00:00     
2023-01-11T21:11:05.4953226Z Running transaction check
2023-01-11T21:11:05.5126098Z Running transaction test
2023-01-11T21:11:05.5291701Z Transaction test succeeded
2023-01-11T21:11:05.5296176Z Running transaction
2023-01-11T21:11:06.0300917Z   Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64                1/5 
2023-01-11T21:11:06.0711272Z   Installing : libnvidia-container1-1.11.0-1.x86_64                         2/5 
2023-01-11T21:11:06.1870559Z   Installing : libnvidia-container-tools-1.11.0-1.x86_64                    3/5 
2023-01-11T21:11:06.2223806Z   Installing : nvidia-container-toolkit-1.11.0-1.x86_64                     4/5 
2023-01-11T21:11:06.2669672Z   Installing : nvidia-docker2-2.11.0-1.noarch                               5/5 
2023-01-11T21:11:06.2820328Z   Verifying  : libnvidia-container1-1.11.0-1.x86_64                         1/5 
2023-01-11T21:11:06.2932221Z   Verifying  : nvidia-container-toolkit-base-1.11.0-1.x86_64                2/5 
2023-01-11T21:11:06.3059760Z   Verifying  : nvidia-container-toolkit-1.11.0-1.x86_64                     3/5 
2023-01-11T21:11:06.3175055Z   Verifying  : libnvidia-container-tools-1.11.0-1.x86_64                    4/5 
2023-01-11T21:11:06.4002480Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               5/5 
2023-01-11T21:11:06.4003019Z 
2023-01-11T21:11:06.4003202Z Installed:
2023-01-11T21:11:06.4003950Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2023-01-11T21:11:06.4004339Z 
2023-01-11T21:11:06.4004554Z Dependency Installed:
2023-01-11T21:11:06.4005348Z   libnvidia-container-tools.x86_64 0:1.11.0-1                                   
2023-01-11T21:11:06.4006267Z   libnvidia-container1.x86_64 0:1.11.0-1                                        
2023-01-11T21:11:06.4007218Z   nvidia-container-toolkit.x86_64 0:1.11.0-1                                    
2023-01-11T21:11:06.4008133Z   nvidia-container-toolkit-base.x86_64 0:1.11.0-1                               
2023-01-11T21:11:06.4008878Z 
2023-01-11T21:11:06.4009049Z Complete!
2023-01-11T21:11:06.5022143Z + sudo systemctl restart docker
2023-01-11T21:11:14.4613552Z Command completed after 1 attempt(s).
2023-01-11T21:11:14.4673197Z ##[group]Run python3 -m pip install psutil==5.9.1
2023-01-11T21:11:14.4673588Z python3 -m pip install psutil==5.9.1
2023-01-11T21:11:14.4673895Z python3 -m pip install pynvml==11.4.1
2023-01-11T21:11:14.4674248Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2023-01-11T21:11:14.4674629Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2023-01-11T21:11:14.4688294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:11:14.4688573Z env:
2023-01-11T21:11:14.4688813Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:11:14.4689079Z   GPU_FLAG: --gpus all
2023-01-11T21:11:14.4689309Z ##[endgroup]
2023-01-11T21:11:15.1090006Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:11:15.4837324Z Collecting psutil==5.9.1
2023-01-11T21:11:15.5028235Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2023-01-11T21:11:15.5754788Z Installing collected packages: psutil
2023-01-11T21:11:15.7328553Z Successfully installed psutil-5.9.1
2023-01-11T21:11:16.2064073Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:11:16.2981711Z Collecting pynvml==11.4.1
2023-01-11T21:11:16.3146398Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2023-01-11T21:11:16.3646747Z Installing collected packages: pynvml
2023-01-11T21:11:16.4204990Z Successfully installed pynvml-11.4.1
2023-01-11T21:11:16.4689938Z Prepare all required actions
2023-01-11T21:11:16.4690335Z Getting action download info
2023-01-11T21:11:16.6320461Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2023-01-11T21:11:16.8205631Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2023-01-11T21:11:16.9433817Z ##[group]Run ./.github/actions/download-build-artifacts
2023-01-11T21:11:16.9434137Z with:
2023-01-11T21:11:16.9434474Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T21:11:16.9434798Z env:
2023-01-11T21:11:16.9435128Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:11:16.9435406Z   GPU_FLAG: --gpus all
2023-01-11T21:11:16.9435647Z ##[endgroup]
2023-01-11T21:11:16.9470575Z ##[group]Run seemethere/download-artifact-s3@v4
2023-01-11T21:11:16.9470868Z with:
2023-01-11T21:11:16.9471132Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T21:11:16.9471440Z   s3-bucket: gha-artifacts
2023-01-11T21:11:16.9471782Z   region: us-east-1
2023-01-11T21:11:16.9471995Z env:
2023-01-11T21:11:16.9472232Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:11:16.9472498Z   GPU_FLAG: --gpus all
2023-01-11T21:11:16.9472728Z ##[endgroup]
2023-01-11T21:11:17.5310576Z Found 1 objects with prefix pytorch/pytorch/3896099317/linux-bionic-cuda11.6-py3.10-gcc7/
2023-01-11T21:11:17.5311213Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T21:11:23.9430593Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T21:11:23.9431198Z 
2023-01-11T21:11:23.9464375Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2023-01-11T21:11:23.9479945Z Artifact download has finished successfully
2023-01-11T21:11:23.9704644Z ##[group]Run unzip -o artifacts.zip
2023-01-11T21:11:23.9704963Z unzip -o artifacts.zip
2023-01-11T21:11:23.9718552Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:11:23.9718843Z env:
2023-01-11T21:11:23.9719081Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:11:23.9719800Z   GPU_FLAG: --gpus all
2023-01-11T21:11:23.9720054Z ##[endgroup]
2023-01-11T21:11:23.9833305Z Archive:  artifacts.zip
2023-01-11T21:11:23.9835168Z    creating: dist/
2023-01-11T21:11:26.0731114Z   inflating: dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl  
2023-01-11T21:11:26.0731581Z    creating: build/custom_test_artifacts/
2023-01-11T21:11:26.0732009Z    creating: build/custom_test_artifacts/custom-op-build/
2023-01-11T21:11:26.0732479Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2023-01-11T21:11:26.0739102Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:11:26.0740162Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2023-01-11T21:11:26.0740744Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:11:26.0741298Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:11:26.0741838Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:11:26.0743178Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:11:26.0744812Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:11:26.0745596Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:11:26.0746139Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:11:26.0748453Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:11:26.0749712Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:11:26.0751316Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:11:26.0752101Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:11:26.0753507Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:11:26.0754764Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:11:26.0755559Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:11:26.0756108Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:11:26.0810548Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:11:26.0811902Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:11:26.0813106Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:11:26.0814539Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:11:26.0815565Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:11:26.0816483Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:11:26.0817170Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:11:26.0817864Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:11:26.0818543Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:11:26.0858344Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:11:26.0899750Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:11:26.0901130Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:11:26.0902221Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:11:26.0903358Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:11:26.0904419Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:11:26.0905263Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:11:26.0906127Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:11:26.0906928Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:11:26.0979856Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:11:26.1052969Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:11:26.1053939Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:11:26.1054966Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2023-01-11T21:11:26.1056169Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2023-01-11T21:11:26.1057125Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:11:26.1057991Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2023-01-11T21:11:26.1058969Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2023-01-11T21:11:26.1059969Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2023-01-11T21:11:26.1061121Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2023-01-11T21:11:26.1061905Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2023-01-11T21:11:26.1062671Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2023-01-11T21:11:26.1063311Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2023-01-11T21:11:26.1063897Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2023-01-11T21:11:26.1064453Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2023-01-11T21:11:26.1065032Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2023-01-11T21:11:26.1083946Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2023-01-11T21:11:26.1201187Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2023-01-11T21:11:26.1202247Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2023-01-11T21:11:26.1203380Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2023-01-11T21:11:26.1204392Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2023-01-11T21:11:26.1205351Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2023-01-11T21:11:26.1206454Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2023-01-11T21:11:26.1207491Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2023-01-11T21:11:26.1208088Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2023-01-11T21:11:26.1208692Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2023-01-11T21:11:26.1209269Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2023-01-11T21:11:26.1209861Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2023-01-11T21:11:26.1228573Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2023-01-11T21:11:26.1315306Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2023-01-11T21:11:26.1316442Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:11:26.1317290Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:11:26.1318026Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2023-01-11T21:11:26.1319006Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2023-01-11T21:11:26.1319872Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:11:26.1320556Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2023-01-11T21:11:26.1323045Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2023-01-11T21:11:26.1324345Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2023-01-11T21:11:26.1324850Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2023-01-11T21:11:26.1419429Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2023-01-11T21:11:26.1484440Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2023-01-11T21:11:26.1484919Z    creating: build/custom_test_artifacts/jit-hook-build/
2023-01-11T21:11:26.1485375Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2023-01-11T21:11:26.1492038Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:11:26.1492935Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2023-01-11T21:11:26.1493531Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:11:26.1494079Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:11:26.1494627Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:11:26.1495915Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:11:26.1497283Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:11:26.1497848Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:11:26.1498410Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:11:26.1501167Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:11:26.1502373Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:11:26.1504354Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:11:26.1504965Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:11:26.1506886Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:11:26.1507699Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:11:26.1508270Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:11:26.1508829Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:11:26.1563482Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:11:26.1564188Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:11:26.1564894Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:11:26.1565610Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:11:26.1566327Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:11:26.1567009Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:11:26.1567670Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:11:26.1568353Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:11:26.1569175Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:11:26.1611261Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:11:26.1652316Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:11:26.1653369Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:11:26.1654108Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:11:26.1654724Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:11:26.1655908Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:11:26.1657038Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:11:26.1657904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:11:26.1659502Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:11:26.1732795Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:11:26.1806083Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:11:26.1807207Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:11:26.1808301Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2023-01-11T21:11:26.1809356Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2023-01-11T21:11:26.1810373Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:11:26.1811424Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2023-01-11T21:11:26.1812268Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2023-01-11T21:11:26.1813716Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2023-01-11T21:11:26.1814819Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2023-01-11T21:11:26.1815384Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2023-01-11T21:11:26.1815966Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2023-01-11T21:11:26.1816542Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2023-01-11T21:11:26.1817138Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2023-01-11T21:11:26.1817711Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2023-01-11T21:11:26.1818286Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2023-01-11T21:11:26.1836072Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2023-01-11T21:11:26.1903239Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2023-01-11T21:11:26.1904323Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:11:26.1905194Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:11:26.1905925Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2023-01-11T21:11:26.1906894Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2023-01-11T21:11:26.1907678Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:11:26.1908199Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2023-01-11T21:11:26.1910486Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2023-01-11T21:11:26.1911232Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2023-01-11T21:11:26.1912038Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2023-01-11T21:11:26.1963446Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2023-01-11T21:11:26.1963928Z    creating: build/custom_test_artifacts/custom-backend-build/
2023-01-11T21:11:26.1964414Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2023-01-11T21:11:26.1971038Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:11:26.1971610Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2023-01-11T21:11:26.1972178Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:11:26.1972761Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:11:26.1973325Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:11:26.1975369Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:11:26.1976504Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:11:26.1977090Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:11:26.1977657Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:11:26.1980247Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:11:26.1981342Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:11:26.1983318Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:11:26.1984102Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:11:26.1985687Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:11:26.1986811Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:11:26.1987410Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:11:26.1987981Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:11:26.2042463Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:11:26.2043206Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:11:26.2043954Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:11:26.2044707Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:11:26.2045432Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:11:26.2046147Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:11:26.2046977Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:11:26.2047705Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:11:26.2048617Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:11:26.2090854Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:11:26.2132041Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:11:26.2133186Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:11:26.2134456Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:11:26.2135642Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:11:26.2136916Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:11:26.2137824Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:11:26.2138784Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:11:26.2139569Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:11:26.2212419Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:11:26.2285463Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:11:26.2286284Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:11:26.2287291Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2023-01-11T21:11:26.2288241Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2023-01-11T21:11:26.2289364Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:11:26.2290240Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2023-01-11T21:11:26.2291250Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2023-01-11T21:11:26.2291997Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2023-01-11T21:11:26.2293059Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2023-01-11T21:11:26.2294063Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2023-01-11T21:11:26.2294936Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2023-01-11T21:11:26.2295724Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2023-01-11T21:11:26.2296354Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2023-01-11T21:11:26.2296951Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2023-01-11T21:11:26.2297567Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2023-01-11T21:11:26.2300480Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2023-01-11T21:11:26.2454366Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2023-01-11T21:11:26.2455634Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2023-01-11T21:11:26.2456770Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2023-01-11T21:11:26.2457692Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2023-01-11T21:11:26.2458855Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2023-01-11T21:11:26.2459958Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2023-01-11T21:11:26.2460799Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2023-01-11T21:11:26.2461436Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2023-01-11T21:11:26.2462079Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2023-01-11T21:11:26.2462688Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2023-01-11T21:11:26.2463365Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2023-01-11T21:11:26.2482158Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2023-01-11T21:11:26.2544805Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2023-01-11T21:11:26.2545952Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:11:26.2546773Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:11:26.2547428Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2023-01-11T21:11:26.2548296Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2023-01-11T21:11:26.2549162Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:11:26.2549715Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2023-01-11T21:11:26.2552041Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2023-01-11T21:11:26.2552899Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2023-01-11T21:11:26.2553926Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2023-01-11T21:11:26.2676469Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2023-01-11T21:11:26.2723938Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2023-01-11T21:11:26.2724276Z    creating: build/lib/
2023-01-11T21:11:26.2724953Z   inflating: build/lib/libclog.a     
2023-01-11T21:11:26.2794862Z   inflating: build/lib/libgtest.a    
2023-01-11T21:11:26.2805385Z   inflating: build/lib/libpthreadpool.a  
2023-01-11T21:11:26.2911321Z   inflating: build/lib/libprotobuf-lite.a  
2023-01-11T21:11:26.3007664Z   inflating: build/lib/libbenchmark.a  
2023-01-11T21:11:26.3016723Z   inflating: build/lib/libittnotify.a  
2023-01-11T21:11:26.3093469Z   inflating: build/lib/libasmjit.a   
2023-01-11T21:11:26.3125426Z   inflating: build/lib/libtensorpipe_uv.a  
2023-01-11T21:11:26.3658435Z   inflating: build/lib/libprotobuf.a  
2023-01-11T21:11:26.3799104Z   inflating: build/lib/libgloo.a     
2023-01-11T21:11:26.3831810Z   inflating: build/lib/libfmt.a      
2023-01-11T21:11:26.3833734Z   inflating: build/lib/libcaffe2_nvrtc.so  
2023-01-11T21:11:26.3834304Z   inflating: build/lib/libfoxi_loader.a  
2023-01-11T21:11:26.3916932Z   inflating: build/lib/libc10.so     
2023-01-11T21:11:26.3918236Z   inflating: build/lib/libtorch_global_deps.so  
2023-01-11T21:11:26.3928581Z   inflating: build/lib/libcpuinfo.a  
2023-01-11T21:11:26.3937504Z   inflating: build/lib/libcpuinfo_internals.a  
2023-01-11T21:11:26.4507166Z   inflating: build/lib/libprotoc.a   
2023-01-11T21:11:26.4523298Z   inflating: build/lib/libqnnpack.a  
2023-01-11T21:11:26.4546230Z   inflating: build/lib/libpytorch_qnnpack.a  
2023-01-11T21:11:26.4548833Z   inflating: build/lib/libnnpack_reference_layers.a  
2023-01-11T21:11:26.4571128Z   inflating: build/lib/libnnpack.a   
2023-01-11T21:11:26.4588725Z   inflating: build/lib/libgmock.a    
2023-01-11T21:11:26.4589336Z   inflating: build/lib/libgtest_main.a  
2023-01-11T21:11:26.4590312Z   inflating: build/lib/libbenchmark_main.a  
2023-01-11T21:11:26.4732231Z   inflating: build/lib/libXNNPACK.a  
2023-01-11T21:11:27.4499877Z   inflating: build/lib/libdnnl.a     
2023-01-11T21:11:27.5154891Z   inflating: build/lib/libtensorpipe.a  
2023-01-11T21:11:27.5209812Z   inflating: build/lib/libc10_cuda.so  
2023-01-11T21:11:27.6751080Z   inflating: build/lib/libfbgemm.a   
2023-01-11T21:11:27.6751666Z   inflating: build/lib/libgmock_main.a  
2023-01-11T21:11:27.7907990Z   inflating: build/lib/libdnnl_graph.a  
2023-01-11T21:11:27.8423834Z   inflating: build/lib/libkineto.a   
2023-01-11T21:11:27.8713453Z   inflating: build/lib/libtensorpipe_cuda.a  
2023-01-11T21:11:27.8758523Z   inflating: build/lib/libcaffe2_protos.a  
2023-01-11T21:11:27.8807324Z   inflating: build/lib/libonnx_proto.a  
2023-01-11T21:11:27.9485446Z   inflating: build/lib/libonnx.a     
2023-01-11T21:11:27.9916943Z   inflating: build/lib/libgloo_cuda.a  
2023-01-11T21:11:30.3706400Z   inflating: build/lib/libtorch_cpu.so  
2023-01-11T21:11:30.3717269Z   inflating: build/lib/libunbox_lib.a  
2023-01-11T21:11:32.5100009Z   inflating: build/lib/libtorch_cuda.so  
2023-01-11T21:11:32.5101089Z   inflating: build/lib/libtorch.so   
2023-01-11T21:11:32.5104203Z   inflating: build/lib/libc10d_cuda_test.so  
2023-01-11T21:11:33.4976632Z   inflating: build/lib/libtorch_cuda_linalg.so  
2023-01-11T21:11:33.5001107Z   inflating: build/lib/libjitbackend_test.so  
2023-01-11T21:11:33.5031916Z   inflating: build/lib/libbackend_with_compiler.so  
2023-01-11T21:11:33.5093158Z   inflating: build/lib/libtorchbind_test.so  
2023-01-11T21:11:33.5098098Z   inflating: build/lib/libshm.so     
2023-01-11T21:11:33.6950531Z   inflating: build/lib/libtorch_python.so  
2023-01-11T21:11:33.6990886Z   inflating: build/lib/libnnapi_backend.so  
2023-01-11T21:11:33.6991311Z    creating: build/bin/
2023-01-11T21:11:33.7045696Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2023-01-11T21:11:33.7103208Z   inflating: build/bin/c10_DeviceGuard_test  
2023-01-11T21:11:33.7159191Z   inflating: build/bin/c10_Device_test  
2023-01-11T21:11:33.7224042Z   inflating: build/bin/c10_DispatchKeySet_test  
2023-01-11T21:11:33.7277096Z   inflating: build/bin/c10_StreamGuard_test  
2023-01-11T21:11:33.7331812Z   inflating: build/bin/c10_SymInt_test  
2023-01-11T21:11:33.7393342Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2023-01-11T21:11:33.7455808Z   inflating: build/bin/c10_InlineStreamGuard_test  
2023-01-11T21:11:33.7518451Z   inflating: build/bin/c10_SizesAndStrides_test  
2023-01-11T21:11:33.7571808Z   inflating: build/bin/c10_Array_test  
2023-01-11T21:11:33.7630846Z   inflating: build/bin/c10_Bitset_test  
2023-01-11T21:11:33.7687818Z   inflating: build/bin/c10_C++17_test  
2023-01-11T21:11:33.7740759Z   inflating: build/bin/c10_ConstexprCrc_test  
2023-01-11T21:11:33.7795025Z   inflating: build/bin/c10_DeadlockDetection_test  
2023-01-11T21:11:33.7850139Z   inflating: build/bin/c10_Half_test  
2023-01-11T21:11:33.7912747Z   inflating: build/bin/c10_LeftRight_test  
2023-01-11T21:11:33.7981920Z   inflating: build/bin/c10_Metaprogramming_test  
2023-01-11T21:11:33.8142536Z   inflating: build/bin/c10_SmallVectorTest  
2023-01-11T21:11:33.8197965Z   inflating: build/bin/c10_Synchronized_test  
2023-01-11T21:11:33.8260937Z   inflating: build/bin/c10_ThreadLocal_test  
2023-01-11T21:11:33.8319550Z   inflating: build/bin/c10_TypeIndex_test  
2023-01-11T21:11:33.8375428Z   inflating: build/bin/c10_TypeList_test  
2023-01-11T21:11:33.8428654Z   inflating: build/bin/c10_TypeTraits_test  
2023-01-11T21:11:33.8486661Z   inflating: build/bin/c10_accumulate_test  
2023-01-11T21:11:33.8548923Z   inflating: build/bin/c10_bfloat16_test  
2023-01-11T21:11:33.8610189Z   inflating: build/bin/c10_complex_math_test  
2023-01-11T21:11:33.8671047Z   inflating: build/bin/c10_complex_test  
2023-01-11T21:11:33.8790700Z   inflating: build/bin/c10_either_test  
2023-01-11T21:11:33.8849378Z   inflating: build/bin/c10_exception_test  
2023-01-11T21:11:33.8904986Z   inflating: build/bin/c10_flags_test  
2023-01-11T21:11:33.9089744Z   inflating: build/bin/c10_intrusive_ptr_test  
2023-01-11T21:11:33.9145662Z   inflating: build/bin/c10_irange_test  
2023-01-11T21:11:33.9208980Z   inflating: build/bin/c10_logging_test  
2023-01-11T21:11:33.9290413Z   inflating: build/bin/c10_optional_test  
2023-01-11T21:11:33.9358735Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2023-01-11T21:11:33.9419771Z   inflating: build/bin/c10_registry_test  
2023-01-11T21:11:33.9484024Z   inflating: build/bin/c10_string_view_test  
2023-01-11T21:11:33.9541212Z   inflating: build/bin/c10_tempfile_test  
2023-01-11T21:11:33.9603103Z   inflating: build/bin/c10_typeid_test  
2023-01-11T21:11:33.9664001Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2023-01-11T21:11:34.0185523Z   inflating: build/bin/protoc-3.13.0.0  
2023-01-11T21:11:34.0706529Z   inflating: build/bin/protoc        
2023-01-11T21:11:34.0765465Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2023-01-11T21:11:34.0824607Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2023-01-11T21:11:34.0883386Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2023-01-11T21:11:34.0941242Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2023-01-11T21:11:34.1000195Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2023-01-11T21:11:34.1053925Z   inflating: build/bin/c10_cuda_CUDATest  
2023-01-11T21:11:34.1113276Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2023-01-11T21:11:34.1172154Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2023-01-11T21:11:34.1495637Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2023-01-11T21:11:34.1855205Z   inflating: build/bin/vec_test_all_types_AVX2  
2023-01-11T21:11:34.1914831Z   inflating: build/bin/FileStoreTest  
2023-01-11T21:11:34.1980925Z   inflating: build/bin/TCPStoreTest  
2023-01-11T21:11:34.2040867Z   inflating: build/bin/HashStoreTest  
2023-01-11T21:11:34.2057105Z   inflating: build/bin/ProcessGroupMPITest  
2023-01-11T21:11:34.2120062Z   inflating: build/bin/test_edge_op_registration  
2023-01-11T21:11:34.2123543Z   inflating: build/bin/example_allreduce  
2023-01-11T21:11:34.2182227Z   inflating: build/bin/Dimname_test  
2023-01-11T21:11:34.2262669Z   inflating: build/bin/Dict_test     
2023-01-11T21:11:34.2333458Z   inflating: build/bin/MaybeOwned_test  
2023-01-11T21:11:34.2396566Z   inflating: build/bin/NamedTensor_test  
2023-01-11T21:11:34.2462366Z   inflating: build/bin/apply_utils_test  
2023-01-11T21:11:34.2527309Z   inflating: build/bin/atest         
2023-01-11T21:11:34.2594341Z   inflating: build/bin/basic         
2023-01-11T21:11:34.2654229Z   inflating: build/bin/broadcast_test  
2023-01-11T21:11:34.2718141Z   inflating: build/bin/cpu_generator_test  
2023-01-11T21:11:34.2773504Z   inflating: build/bin/dispatch_key_set_test  
2023-01-11T21:11:34.2831629Z   inflating: build/bin/cpu_profiling_allocator_test  
2023-01-11T21:11:34.2886585Z   inflating: build/bin/dlconvertor_test  
2023-01-11T21:11:34.2950920Z   inflating: build/bin/extension_backend_test  
2023-01-11T21:11:34.3012522Z   inflating: build/bin/half_test     
2023-01-11T21:11:34.3067011Z   inflating: build/bin/lazy_tensor_test  
2023-01-11T21:11:34.3127165Z   inflating: build/bin/math_kernel_test  
2023-01-11T21:11:34.3186650Z   inflating: build/bin/memory_format_test  
2023-01-11T21:11:34.3246342Z   inflating: build/bin/memory_overlapping_test  
2023-01-11T21:11:34.3302512Z   inflating: build/bin/operator_name_test  
2023-01-11T21:11:34.3364313Z   inflating: build/bin/native_test   
2023-01-11T21:11:34.3419853Z   inflating: build/bin/operators_test  
2023-01-11T21:11:34.3478492Z   inflating: build/bin/packedtensoraccessor_test  
2023-01-11T21:11:34.3542349Z   inflating: build/bin/quantized_test  
2023-01-11T21:11:34.3598794Z   inflating: build/bin/reportMemoryUsage_test  
2023-01-11T21:11:34.3661672Z   inflating: build/bin/scalar_tensor_test  
2023-01-11T21:11:34.3725089Z   inflating: build/bin/scalar_test   
2023-01-11T21:11:34.3783373Z   inflating: build/bin/stride_properties_test  
2023-01-11T21:11:34.3869557Z   inflating: build/bin/tensor_iterator_test  
2023-01-11T21:11:34.3925460Z   inflating: build/bin/reduce_ops_test  
2023-01-11T21:11:34.3998013Z   inflating: build/bin/pow_test      
2023-01-11T21:11:34.4001575Z   inflating: build/bin/thread_init_test  
2023-01-11T21:11:34.4063608Z   inflating: build/bin/test_parallel  
2023-01-11T21:11:34.4124818Z   inflating: build/bin/type_ptr_test  
2023-01-11T21:11:34.4179345Z   inflating: build/bin/variant_test  
2023-01-11T21:11:34.4238013Z   inflating: build/bin/undefined_tensor_test  
2023-01-11T21:11:34.4296193Z   inflating: build/bin/mobile_memory_cleanup  
2023-01-11T21:11:34.4297325Z   inflating: build/bin/verify_api_visibility  
2023-01-11T21:11:34.4373858Z   inflating: build/bin/legacy_vmap_test  
2023-01-11T21:11:34.4430341Z   inflating: build/bin/weakref_test  
2023-01-11T21:11:34.4487060Z   inflating: build/bin/wrapdim_test  
2023-01-11T21:11:34.4541098Z   inflating: build/bin/xla_tensor_test  
2023-01-11T21:11:34.4607552Z   inflating: build/bin/IListRef_test  
2023-01-11T21:11:34.4727247Z   inflating: build/bin/List_test     
2023-01-11T21:11:34.4798751Z   inflating: build/bin/KernelFunction_test  
2023-01-11T21:11:34.4932592Z   inflating: build/bin/kernel_function_legacy_test  
2023-01-11T21:11:34.5038298Z   inflating: build/bin/kernel_function_test  
2023-01-11T21:11:34.5135360Z   inflating: build/bin/cpu_rng_test  
2023-01-11T21:11:34.5276312Z   inflating: build/bin/kernel_lambda_legacy_test  
2023-01-11T21:11:34.5380058Z   inflating: build/bin/ivalue_test   
2023-01-11T21:11:34.5494293Z   inflating: build/bin/kernel_lambda_test  
2023-01-11T21:11:34.5560964Z   inflating: build/bin/kernel_stackbased_test  
2023-01-11T21:11:34.5617535Z   inflating: build/bin/CppSignature_test  
2023-01-11T21:11:34.5722460Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2023-01-11T21:11:34.5775726Z   inflating: build/bin/op_allowlist_test  
2023-01-11T21:11:34.5843009Z   inflating: build/bin/type_test     
2023-01-11T21:11:34.5905264Z   inflating: build/bin/backend_fallback_test  
2023-01-11T21:11:34.6220229Z   inflating: build/bin/op_registration_test  
2023-01-11T21:11:34.6280055Z   inflating: build/bin/inline_container_test  
2023-01-11T21:11:34.6338887Z   inflating: build/bin/cuda_apply_test  
2023-01-11T21:11:34.6405422Z   inflating: build/bin/cuda_atomic_ops_test  
2023-01-11T21:11:34.6465029Z   inflating: build/bin/cuda_caching_host_allocator_test  
2023-01-11T21:11:34.6543236Z   inflating: build/bin/cuda_complex_math_test  
2023-01-11T21:11:34.6608170Z   inflating: build/bin/cuda_complex_test  
2023-01-11T21:11:34.6662496Z   inflating: build/bin/cuda_device_test  
2023-01-11T21:11:34.6728142Z   inflating: build/bin/cuda_cub_test  
2023-01-11T21:11:34.6783441Z   inflating: build/bin/cuda_dlconvertor_test  
2023-01-11T21:11:34.6857465Z   inflating: build/bin/cuda_distributions_test  
2023-01-11T21:11:34.6913248Z   inflating: build/bin/cuda_integer_divider_test  
2023-01-11T21:11:34.6978212Z   inflating: build/bin/cuda_generator_test  
2023-01-11T21:11:34.7032537Z   inflating: build/bin/cuda_half_test  
2023-01-11T21:11:34.7099705Z   inflating: build/bin/cuda_stream_test  
2023-01-11T21:11:34.7158119Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2023-01-11T21:11:34.7212003Z   inflating: build/bin/cuda_optional_test  
2023-01-11T21:11:34.7268800Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2023-01-11T21:11:34.7322663Z   inflating: build/bin/cuda_cudnn_test  
2023-01-11T21:11:34.7380611Z   inflating: build/bin/cuda_vectorized_test  
2023-01-11T21:11:34.7398500Z   inflating: build/bin/tutorial_tensorexpr  
2023-01-11T21:11:34.7470871Z   inflating: build/bin/ProcessGroupGlooTest  
2023-01-11T21:11:34.7535451Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2023-01-11T21:11:34.7603497Z   inflating: build/bin/ProcessGroupNCCLTest  
2023-01-11T21:11:34.7667435Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2023-01-11T21:11:34.7726338Z   inflating: build/bin/ProcessGroupUCCTest  
2023-01-11T21:11:34.7786778Z   inflating: build/bin/test_dist_autograd  
2023-01-11T21:11:34.7863630Z   inflating: build/bin/test_cpp_rpc  
2023-01-11T21:11:34.7866264Z   inflating: build/bin/parallel_benchmark  
2023-01-11T21:11:34.7942110Z   inflating: build/bin/test_mobile_nnc  
2023-01-11T21:11:34.7953466Z   inflating: build/bin/aot_model_compiler_test  
2023-01-11T21:11:34.8862073Z   inflating: build/bin/test_tensorexpr  
2023-01-11T21:11:34.8867624Z   inflating: build/bin/torch_shm_manager  
2023-01-11T21:11:34.9256331Z   inflating: build/bin/test_lazy     
2023-01-11T21:11:35.0575465Z   inflating: build/bin/test_api      
2023-01-11T21:11:35.1778235Z   inflating: build/bin/test_jit      
2023-01-11T21:11:35.1780616Z   inflating: .pytorch-test-times.json  
2023-01-11T21:11:35.1811261Z ##[group]Run df -H
2023-01-11T21:11:35.1811499Z df -H
2023-01-11T21:11:35.1825050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:11:35.1825348Z env:
2023-01-11T21:11:35.1825592Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:11:35.1825841Z   GPU_FLAG: --gpus all
2023-01-11T21:11:35.1826095Z ##[endgroup]
2023-01-11T21:11:35.1865786Z Filesystem      Size  Used Avail Use% Mounted on
2023-01-11T21:11:35.1866102Z devtmpfs        129G     0  129G   0% /dev
2023-01-11T21:11:35.1866385Z tmpfs           129G     0  129G   0% /dev/shm
2023-01-11T21:11:35.1866646Z tmpfs           129G  607k  129G   1% /run
2023-01-11T21:11:35.1867087Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2023-01-11T21:11:35.1867611Z /dev/xvda1      162G   29G  133G  18% /
2023-01-11T21:11:35.1867900Z tmpfs            26G     0   26G   0% /run/user/0
2023-01-11T21:11:35.1893498Z ##[group]Run .github/scripts/parse_ref.py
2023-01-11T21:11:35.1893869Z .github/scripts/parse_ref.py
2023-01-11T21:11:35.1906020Z shell: /usr/bin/bash -e {0}
2023-01-11T21:11:35.1906269Z env:
2023-01-11T21:11:35.1906492Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:11:35.1906765Z   GPU_FLAG: --gpus all
2023-01-11T21:11:35.1907013Z ##[endgroup]
2023-01-11T21:11:35.2204990Z ##[group]Run set -x
2023-01-11T21:11:35.2205413Z set -x
2023-01-11T21:11:35.2205643Z 
2023-01-11T21:11:35.2205896Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2023-01-11T21:11:35.2206242Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2023-01-11T21:11:35.2206586Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2023-01-11T21:11:35.2206914Z   TEST_COMMAND=.jenkins/onnx/test.sh
2023-01-11T21:11:35.2207162Z else
2023-01-11T21:11:35.2207434Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T21:11:35.2207703Z fi
2023-01-11T21:11:35.2207903Z 
2023-01-11T21:11:35.2208222Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2023-01-11T21:11:35.2208534Z 
2023-01-11T21:11:35.2208810Z # sanitize the input commit message and PR body here:
2023-01-11T21:11:35.2209097Z #
2023-01-11T21:11:35.2209471Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2023-01-11T21:11:35.2209955Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2023-01-11T21:11:35.2210377Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2023-01-11T21:11:35.2210687Z PR_BODY="${PR_BODY//[$'\n\r']}"
2023-01-11T21:11:35.2210937Z 
2023-01-11T21:11:35.2211276Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2023-01-11T21:11:35.2211649Z # wreak havoc internally
2023-01-11T21:11:35.2211966Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2023-01-11T21:11:35.2212277Z export PR_BODY="${PR_BODY//[\'\"]}"
2023-01-11T21:11:35.2212532Z 
2023-01-11T21:11:35.2212836Z # detached container should get cleaned up by teardown_ec2_linux
2023-01-11T21:11:35.2213215Z # TODO: Stop building test binaries as part of the build phase
2023-01-11T21:11:35.2213583Z # Used for GPU_FLAG since that doesn't play nice
2023-01-11T21:11:35.2213904Z # shellcheck disable=SC2086,SC2090
2023-01-11T21:11:35.2214200Z container_name=$(docker run \
2023-01-11T21:11:35.2214456Z   ${GPU_FLAG:-} \
2023-01-11T21:11:35.2214723Z   -e BUILD_ENVIRONMENT \
2023-01-11T21:11:35.2214993Z   -e PR_NUMBER \
2023-01-11T21:11:35.2215243Z   -e GITHUB_ACTIONS \
2023-01-11T21:11:35.2215500Z   -e BASE_SHA \
2023-01-11T21:11:35.2215746Z   -e BRANCH \
2023-01-11T21:11:35.2215968Z   -e SHA1 \
2023-01-11T21:11:35.2216220Z   -e AWS_DEFAULT_REGION \
2023-01-11T21:11:35.2216487Z   -e IN_WHEEL_TEST \
2023-01-11T21:11:35.2216729Z   -e SHARD_NUMBER \
2023-01-11T21:11:35.2216988Z   -e TEST_CONFIG \
2023-01-11T21:11:35.2217247Z   -e NUM_TEST_SHARDS \
2023-01-11T21:11:35.2217502Z   -e PR_BODY \
2023-01-11T21:11:35.2217748Z   -e COMMIT_MESSAGES \
2023-01-11T21:11:35.2218024Z   -e CONTINUE_THROUGH_ERROR \
2023-01-11T21:11:35.2218320Z   -e PYTORCH_RETRY_TEST_CASES \
2023-01-11T21:11:35.2218612Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2023-01-11T21:11:35.2218895Z   -e PR_LABELS \
2023-01-11T21:11:35.2219179Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2023-01-11T21:11:35.2219562Z   -e SCCACHE_BUCKET \
2023-01-11T21:11:35.2219836Z   -e SCCACHE_S3_KEY_PREFIX \
2023-01-11T21:11:35.2220104Z   -e XLA_CUDA \
2023-01-11T21:11:35.2220369Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2023-01-11T21:11:35.2220686Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2023-01-11T21:11:35.2221012Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2023-01-11T21:11:35.2221358Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2023-01-11T21:11:35.2221662Z   --ulimit stack=10485760:83886080 \
2023-01-11T21:11:35.2222027Z   --security-opt seccomp=unconfined \
2023-01-11T21:11:35.2222341Z   --cap-add=SYS_PTRACE \
2023-01-11T21:11:35.2222593Z   --ipc=host \
2023-01-11T21:11:35.2222857Z   --shm-size="${SHM_SIZE}" \
2023-01-11T21:11:35.2223113Z   --tty \
2023-01-11T21:11:35.2223334Z   --detach \
2023-01-11T21:11:35.2223597Z   --name="${container_name}" \
2023-01-11T21:11:35.2223873Z   --user jenkins \
2023-01-11T21:11:35.2224174Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2023-01-11T21:11:35.2224508Z   -w /var/lib/jenkins/workspace \
2023-01-11T21:11:35.2224787Z   "${DOCKER_IMAGE}"
2023-01-11T21:11:35.2225027Z )
2023-01-11T21:11:35.2225308Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2023-01-11T21:11:35.2306033Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2023-01-11T21:11:35.2318331Z shell: /usr/bin/bash -e {0}
2023-01-11T21:11:35.2318592Z env:
2023-01-11T21:11:35.2318827Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:11:35.2319105Z   GPU_FLAG: --gpus all
2023-01-11T21:11:35.2319676Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T21:11:35.2320004Z   PR_NUMBER: 91627
2023-01-11T21:11:35.2320260Z   BRANCH: pull/91627
2023-01-11T21:11:35.2320566Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:11:35.2320900Z   BASE_SHA: db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T21:11:35.2321210Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T21:11:35.2321510Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T21:11:35.2321782Z   TEST_CONFIG: distributed
2023-01-11T21:11:35.2322039Z   SHARD_NUMBER: 2
2023-01-11T21:11:35.2322290Z   NUM_TEST_SHARDS: 3
2023-01-11T21:11:35.2322569Z   PR_BODY: Fixes #91003 


cc @ezyang @gchanan
2023-01-11T21:11:35.2322880Z   CONTINUE_THROUGH_ERROR: False
2023-01-11T21:11:35.2323229Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2023-01-11T21:11:35.2323571Z   SCCACHE_S3_KEY_PREFIX: pull
2023-01-11T21:11:35.2323822Z   SHM_SIZE: 2g
2023-01-11T21:11:35.2324333Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:11:35.2324827Z   XLA_CUDA: 
2023-01-11T21:11:35.2325172Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:11:35.2325577Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2023-01-11T21:11:35.2325892Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2023-01-11T21:11:35.2326155Z ##[endgroup]
2023-01-11T21:11:35.2355290Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2023-01-11T21:11:35.2355764Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2023-01-11T21:11:35.2356098Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T21:11:35.2359258Z ++ git cherry -v origin/master
2023-01-11T21:11:35.2875601Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''
2023-01-11T21:11:35.2876078Z + 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch
2023-01-11T21:11:35.2876644Z + 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T21:11:35.2877814Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T21:11:35.2878663Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan'
2023-01-11T21:11:35.2880301Z + export 'COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T21:11:35.2881542Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T21:11:35.2882224Z + export 'PR_BODY=Fixes #91003 cc @ezyang @gchanan'
2023-01-11T21:11:35.2882584Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan'
2023-01-11T21:11:35.2890650Z +++ nproc --ignore=2
2023-01-11T21:11:35.2932059Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3896099317 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:11:49.6713655Z + container_name=f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T21:11:49.6714186Z + echo DOCKER_CONTAINER_ID=f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T21:11:49.6719086Z ++ echo dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl
2023-01-11T21:11:49.6721729Z + docker exec -t f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc sh -c 'pip install dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2023-01-11T21:11:50.2288460Z Processing ./dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl
2023-01-11T21:11:51.1908443Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (2.6.3)
2023-01-11T21:11:51.1911204Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (4.4.0)
2023-01-11T21:11:51.1915513Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (1.11.1)
2023-01-11T21:11:51.1932345Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (3.3.0)
2023-01-11T21:11:51.2015320Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.0.0a0+git8419ddd) (1.21.2)
2023-01-11T21:11:51.2229422Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==2.0.0a0+git8419ddd) (1.2.1)
2023-01-11T21:11:52.1522420Z Installing collected packages: torch
2023-01-11T21:12:01.7353577Z Successfully installed torch-2.0.0a0+git8419ddd
2023-01-11T21:12:01.8934818Z + echo 'Environment variables:'
2023-01-11T21:12:01.8935350Z Environment variables:
2023-01-11T21:12:01.8935584Z + env
2023-01-11T21:12:01.8942434Z SHARD_NUMBER=2
2023-01-11T21:12:01.8942962Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2023-01-11T21:12:01.8943356Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2023-01-11T21:12:01.8944271Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-01-11T21:12:01.8945787Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2023-01-11T21:12:01.8946091Z UCC_HOME=/usr
2023-01-11T21:12:01.8946609Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T21:12:01.8947094Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T21:12:01.8947504Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2023-01-11T21:12:01.8947864Z INSTALLED_DB=yes
2023-01-11T21:12:01.8948309Z HOSTNAME=f50ab5966376
2023-01-11T21:12:01.8948745Z GITHUB_REF_NAME=91627/merge
2023-01-11T21:12:01.8949088Z GITHUB_API_URL=https://api.github.com
2023-01-11T21:12:01.8949655Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T21:12:01.8949962Z OPENSSL_DIR=/opt/openssl
2023-01-11T21:12:01.8950274Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2023-01-11T21:12:01.8951154Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.8951579Z CUDA_PATH=/usr/local/cuda
2023-01-11T21:12:01.8952254Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T21:12:01.8952729Z GITHUB_RUN_ATTEMPT=2
2023-01-11T21:12:01.8952973Z TEST_CONFIG=distributed
2023-01-11T21:12:01.8953293Z NV_LIBNPP_VERSION=11.6.3.124-1
2023-01-11T21:12:01.8953933Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2023-01-11T21:12:01.8954268Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T21:12:01.8954527Z GITHUB_ACTIONS=true
2023-01-11T21:12:01.8954908Z NVIDIA_VISIBLE_DEVICES=all
2023-01-11T21:12:01.8955376Z NV_NVPROF_VERSION=11.6.124-1
2023-01-11T21:12:01.8955670Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2023-01-11T21:12:01.8956058Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge
2023-01-11T21:12:01.8956668Z NVIDIA_PRODUCT_NAME=CUDA
2023-01-11T21:12:01.8956900Z CI=true
2023-01-11T21:12:01.8957150Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T21:12:01.8957731Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2023-01-11T21:12:01.8958146Z BRANCH=pull/91627
2023-01-11T21:12:01.8958402Z GITHUB_HEAD_REF=master
2023-01-11T21:12:01.8958719Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2023-01-11T21:12:01.8959222Z GITHUB_ACTOR=LucaLumetti
2023-01-11T21:12:01.8960090Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-01-11T21:12:01.8960643Z GITHUB_ACTION_REF=
2023-01-11T21:12:01.8961015Z NCCL_VERSION=2.12.10-1
2023-01-11T21:12:01.8964834Z GITHUB_ACTION=__self
2023-01-11T21:12:01.8965226Z GITHUB_REF_PROTECTED=false
2023-01-11T21:12:01.8966114Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:12:01.8966559Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T21:12:01.8969504Z ***
2023-01-11T21:12:01.8969781Z INSTALLED_VISION=yes
2023-01-11T21:12:01.8970035Z NVARCH=x86_64
2023-01-11T21:12:01.8970358Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2023-01-11T21:12:01.8970645Z HOME=/var/lib/jenkins
2023-01-11T21:12:01.8971176Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.8971587Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T21:12:01.8971852Z GITHUB_ACTION_REPOSITORY=
2023-01-11T21:12:01.8972117Z GITHUB_REF_TYPE=branch
2023-01-11T21:12:01.8972428Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2023-01-11T21:12:01.8972695Z GITHUB_RETENTION_DAYS=90
2023-01-11T21:12:01.8973078Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T21:12:01.8973490Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2023-01-11T21:12:01.8974020Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.8974418Z DEBIAN_FRONTEND=noninteractive
2023-01-11T21:12:01.8974768Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-01-11T21:12:01.8975052Z GITHUB_REF=refs/pull/91627/merge
2023-01-11T21:12:01.8975363Z NV_CUDA_LIB_VERSION=11.6.2-1
2023-01-11T21:12:01.8975684Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T21:12:01.8976159Z INSTALLED_PROTOBUF=yes
2023-01-11T21:12:01.8976444Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T21:12:01.8976805Z GITHUB_RUN_ID=3896099317
2023-01-11T21:12:01.8977185Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2023-01-11T21:12:01.8977492Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-01-11T21:12:01.8977807Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-01-11T21:12:01.8978133Z NV_NVTX_VERSION=11.6.124-1
2023-01-11T21:12:01.8978402Z CONTINUE_THROUGH_ERROR=False
2023-01-11T21:12:01.8978725Z GITHUB_SERVER_URL=https://github.com
2023-01-11T21:12:01.8979112Z MAX_JOBS=30
2023-01-11T21:12:01.8979360Z GITHUB_ACTOR_ID=7543386
2023-01-11T21:12:01.8979679Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2023-01-11T21:12:01.8980078Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2023-01-11T21:12:01.8980575Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T21:12:01.8980941Z UCX_HOME=/usr
2023-01-11T21:12:01.8981210Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T21:12:01.8981541Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T21:12:01.8981908Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T21:12:01.8982264Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2023-01-11T21:12:01.8982579Z PR_BODY=Fixes #91003 cc @ezyang @gchanan
2023-01-11T21:12:01.8982850Z GITHUB_BASE_REF=master
2023-01-11T21:12:01.8983101Z TERM=xterm
2023-01-11T21:12:01.8983331Z XLA_CUDA=
2023-01-11T21:12:01.8983599Z NV_NVML_DEV_VERSION=11.6.55-1
2023-01-11T21:12:01.8983887Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T21:12:01.8984157Z CUDA_VERSION=11.6.2
2023-01-11T21:12:01.8984504Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2023-01-11T21:12:01.8984821Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T21:12:01.8985391Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.8985777Z GITHUB_JOB=test
2023-01-11T21:12:01.8986044Z SCCACHE_S3_KEY_PREFIX=pull
2023-01-11T21:12:01.8986674Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T21:12:01.8987398Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2023-01-11T21:12:01.8987681Z NUM_TEST_SHARDS=3
2023-01-11T21:12:01.8987931Z PR_NUMBER=91627
2023-01-11T21:12:01.8988491Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.8988869Z SHLVL=1
2023-01-11T21:12:01.8989235Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2023-01-11T21:12:01.8989578Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T21:12:01.8990438Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471
2023-01-11T21:12:01.8991272Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2023-01-11T21:12:01.8991587Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:12:01.8991901Z GITHUB_EVENT_NAME=pull_request
2023-01-11T21:12:01.8992232Z NV_CUDA_CUDART_VERSION=11.6.55-1
2023-01-11T21:12:01.8992581Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T21:12:01.8992881Z GITHUB_RUN_NUMBER=77928
2023-01-11T21:12:01.8993155Z GITHUB_WORKFLOW=pull
2023-01-11T21:12:01.8993580Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:12:01.8994062Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2023-01-11T21:12:01.8994418Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T21:12:01.8994882Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:12:01.8995332Z GITHUB_TRIGGERING_ACTOR=albanD
2023-01-11T21:12:01.8995601Z _=/usr/bin/env
2023-01-11T21:12:01.8995991Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2023-01-11T21:12:01.9174785Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2023-01-11T21:12:01.9175478Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T21:12:01.9175930Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2023-01-11T21:12:01.9176756Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2023-01-11T21:12:01.9177107Z + BUILD_DIR=build
2023-01-11T21:12:01.9177363Z + BUILD_RENAMED_DIR=build_renamed
2023-01-11T21:12:01.9177629Z + BUILD_BIN_DIR=build/bin
2023-01-11T21:12:01.9177888Z + export VALGRIND=ON
2023-01-11T21:12:01.9178131Z + VALGRIND=ON
2023-01-11T21:12:01.9178390Z + export TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:12:01.9178694Z + TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:12:01.9179097Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]]
2023-01-11T21:12:01.9179745Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2023-01-11T21:12:01.9180319Z ++ realpath build/custom_test_artifacts
2023-01-11T21:12:01.9188605Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2023-01-11T21:12:01.9192143Z ++ dirname .jenkins/pytorch/test.sh
2023-01-11T21:12:01.9198642Z + source .jenkins/pytorch/common.sh
2023-01-11T21:12:01.9203806Z +++ dirname .jenkins/pytorch/common.sh
2023-01-11T21:12:01.9213991Z ++ source .jenkins/pytorch/common_utils.sh
2023-01-11T21:12:01.9216180Z +++ declare -f -t trap_add
2023-01-11T21:12:01.9222372Z ++ set -ex
2023-01-11T21:12:01.9222740Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:12:01.9223062Z ++ BUILD_TEST_LIBTORCH=0
2023-01-11T21:12:01.9223369Z + echo 'Environment variables'
2023-01-11T21:12:01.9223647Z Environment variables
2023-01-11T21:12:01.9223883Z + env
2023-01-11T21:12:01.9230978Z SHARD_NUMBER=2
2023-01-11T21:12:01.9231335Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2023-01-11T21:12:01.9231700Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2023-01-11T21:12:01.9232072Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-01-11T21:12:01.9232542Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2023-01-11T21:12:01.9233000Z UCC_HOME=/usr
2023-01-11T21:12:01.9233582Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T21:12:01.9233951Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T21:12:01.9234348Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2023-01-11T21:12:01.9234653Z INSTALLED_DB=yes
2023-01-11T21:12:01.9234880Z HOSTNAME=f50ab5966376
2023-01-11T21:12:01.9235144Z GITHUB_REF_NAME=91627/merge
2023-01-11T21:12:01.9235556Z GITHUB_API_URL=https://api.github.com
2023-01-11T21:12:01.9236019Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T21:12:01.9236312Z OPENSSL_DIR=/opt/openssl
2023-01-11T21:12:01.9236758Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2023-01-11T21:12:01.9237858Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.9238530Z CUDA_PATH=/usr/local/cuda
2023-01-11T21:12:01.9239353Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T21:12:01.9240014Z GITHUB_RUN_ATTEMPT=2
2023-01-11T21:12:01.9240251Z TEST_CONFIG=distributed
2023-01-11T21:12:01.9240566Z NV_LIBNPP_VERSION=11.6.3.124-1
2023-01-11T21:12:01.9240955Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2023-01-11T21:12:01.9241278Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T21:12:01.9241562Z GITHUB_ACTIONS=true
2023-01-11T21:12:01.9241832Z NVIDIA_VISIBLE_DEVICES=all
2023-01-11T21:12:01.9242128Z NV_NVPROF_VERSION=11.6.124-1
2023-01-11T21:12:01.9242454Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2023-01-11T21:12:01.9242851Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge
2023-01-11T21:12:01.9243210Z NVIDIA_PRODUCT_NAME=CUDA
2023-01-11T21:12:01.9243610Z CI=true
2023-01-11T21:12:01.9243867Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T21:12:01.9244298Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2023-01-11T21:12:01.9244598Z BRANCH=pull/91627
2023-01-11T21:12:01.9244861Z GITHUB_HEAD_REF=master
2023-01-11T21:12:01.9245186Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2023-01-11T21:12:01.9245494Z GITHUB_ACTOR=LucaLumetti
2023-01-11T21:12:01.9245817Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-01-11T21:12:01.9246122Z GITHUB_ACTION_REF=
2023-01-11T21:12:01.9246470Z NCCL_VERSION=2.12.10-1
2023-01-11T21:12:01.9246801Z GITHUB_ACTION=__self
2023-01-11T21:12:01.9247053Z VALGRIND=ON
2023-01-11T21:12:01.9247313Z GITHUB_REF_PROTECTED=false
2023-01-11T21:12:01.9247774Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:12:01.9248174Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T21:12:01.9248535Z ***
2023-01-11T21:12:01.9248758Z INSTALLED_VISION=yes
2023-01-11T21:12:01.9249018Z NVARCH=x86_64
2023-01-11T21:12:01.9249334Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2023-01-11T21:12:01.9249604Z HOME=/var/lib/jenkins
2023-01-11T21:12:01.9250152Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.9250577Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T21:12:01.9250868Z GITHUB_ACTION_REPOSITORY=
2023-01-11T21:12:01.9251128Z GITHUB_REF_TYPE=branch
2023-01-11T21:12:01.9251449Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2023-01-11T21:12:01.9251745Z GITHUB_RETENTION_DAYS=90
2023-01-11T21:12:01.9252135Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T21:12:01.9252569Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2023-01-11T21:12:01.9253139Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.9253546Z DEBIAN_FRONTEND=noninteractive
2023-01-11T21:12:01.9253914Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-01-11T21:12:01.9254230Z GITHUB_REF=refs/pull/91627/merge
2023-01-11T21:12:01.9254528Z NV_CUDA_LIB_VERSION=11.6.2-1
2023-01-11T21:12:01.9254843Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T21:12:01.9255146Z INSTALLED_PROTOBUF=yes
2023-01-11T21:12:01.9255412Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T21:12:01.9255686Z GITHUB_RUN_ID=3896099317
2023-01-11T21:12:01.9256052Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2023-01-11T21:12:01.9256353Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-01-11T21:12:01.9256664Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-01-11T21:12:01.9257000Z NV_NVTX_VERSION=11.6.124-1
2023-01-11T21:12:01.9257268Z CONTINUE_THROUGH_ERROR=False
2023-01-11T21:12:01.9257583Z GITHUB_SERVER_URL=https://github.com
2023-01-11T21:12:01.9257865Z MAX_JOBS=30
2023-01-11T21:12:01.9258095Z GITHUB_ACTOR_ID=7543386
2023-01-11T21:12:01.9258410Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2023-01-11T21:12:01.9258805Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2023-01-11T21:12:01.9259315Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T21:12:01.9259662Z UCX_HOME=/usr
2023-01-11T21:12:01.9259923Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T21:12:01.9260268Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T21:12:01.9260615Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T21:12:01.9260969Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2023-01-11T21:12:01.9261280Z PR_BODY=Fixes #91003 cc @ezyang @gchanan
2023-01-11T21:12:01.9261547Z GITHUB_BASE_REF=master
2023-01-11T21:12:01.9261796Z TERM=xterm
2023-01-11T21:12:01.9262058Z TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:12:01.9262304Z XLA_CUDA=
2023-01-11T21:12:01.9262585Z NV_NVML_DEV_VERSION=11.6.55-1
2023-01-11T21:12:01.9262870Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T21:12:01.9263125Z CUDA_VERSION=11.6.2
2023-01-11T21:12:01.9263482Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2023-01-11T21:12:01.9263796Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T21:12:01.9264343Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.9264822Z GITHUB_JOB=test
2023-01-11T21:12:01.9265089Z SCCACHE_S3_KEY_PREFIX=pull
2023-01-11T21:12:01.9265719Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T21:12:01.9266382Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2023-01-11T21:12:01.9266677Z NUM_TEST_SHARDS=3
2023-01-11T21:12:01.9266986Z PR_NUMBER=91627
2023-01-11T21:12:01.9267532Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_84a46a02-5616-4727-9f7b-e4dbcb6b5c11
2023-01-11T21:12:01.9267924Z SHLVL=1
2023-01-11T21:12:01.9268281Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2023-01-11T21:12:01.9268603Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T21:12:01.9269464Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471
2023-01-11T21:12:01.9270298Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2023-01-11T21:12:01.9270624Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:12:01.9270946Z GITHUB_EVENT_NAME=pull_request
2023-01-11T21:12:01.9271255Z NV_CUDA_CUDART_VERSION=11.6.55-1
2023-01-11T21:12:01.9271621Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T21:12:01.9271964Z GITHUB_RUN_NUMBER=77928
2023-01-11T21:12:01.9272217Z GITHUB_WORKFLOW=pull
2023-01-11T21:12:01.9272657Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:12:01.9273140Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2023-01-11T21:12:01.9273475Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T21:12:01.9273957Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:12:01.9274322Z GITHUB_TRIGGERING_ACTOR=albanD
2023-01-11T21:12:01.9274574Z _=/usr/bin/env
2023-01-11T21:12:01.9274873Z + echo 'Testing pytorch'
2023-01-11T21:12:01.9275137Z Testing pytorch
2023-01-11T21:12:01.9275403Z + export LANG=C.UTF-8
2023-01-11T21:12:01.9275678Z + LANG=C.UTF-8
2023-01-11T21:12:01.9275927Z + PR_NUMBER=91627
2023-01-11T21:12:01.9276186Z + [[ distributed == \d\e\f\a\u\l\t ]]
2023-01-11T21:12:01.9276500Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2023-01-11T21:12:01.9276923Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:12:01.9277252Z + [[ distributed == \s\l\o\w ]]
2023-01-11T21:12:01.9277667Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2023-01-11T21:12:01.9278134Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2023-01-11T21:12:01.9278498Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-01-11T21:12:01.9278815Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-01-11T21:12:01.9279123Z + [[ distributed == *crossref* ]]
2023-01-11T21:12:01.9279631Z + [[ distributed == *dynamo* ]]
2023-01-11T21:12:01.9279904Z + [[ distributed == *inductor* ]]
2023-01-11T21:12:01.9280316Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:12:01.9280770Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2023-01-11T21:12:01.9281158Z + pip_install --user ninja==1.10.2
2023-01-11T21:12:01.9281574Z + pip install --progress-bar off --user ninja==1.10.2
2023-01-11T21:12:02.4642377Z Collecting ninja==1.10.2
2023-01-11T21:12:02.4834274Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2023-01-11T21:12:03.3857260Z Installing collected packages: ninja
2023-01-11T21:12:03.3957796Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2023-01-11T21:12:03.3958950Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T21:12:03.4024718Z Successfully installed ninja-1.10.2
2023-01-11T21:12:03.4680124Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:12:03.4681518Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:12:03.4682803Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2023-01-11T21:12:03.4683459Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]]
2023-01-11T21:12:03.4683793Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2023-01-11T21:12:03.4684116Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2023-01-11T21:12:03.4691851Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2023-01-11T21:12:03.4706759Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2023-01-11T21:12:03.4707730Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2023-01-11T21:12:03.4708240Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]]
2023-01-11T21:12:03.4710421Z + cd test
2023-01-11T21:12:03.4711211Z + python -c 'import torch; print(torch.__config__.show())'
2023-01-11T21:12:05.1504103Z PyTorch built with:
2023-01-11T21:12:05.1504607Z   - GCC 7.5
2023-01-11T21:12:05.1504879Z   - C++ Version: 201703
2023-01-11T21:12:05.1505415Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T21:12:05.1505979Z   - Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T21:12:05.1506367Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T21:12:05.1506746Z   - LAPACK is enabled (usually provided by MKL)
2023-01-11T21:12:05.1507073Z   - NNPACK is enabled
2023-01-11T21:12:05.1507391Z   - CPU capability usage: AVX2
2023-01-11T21:12:05.1507679Z   - CUDA Runtime 11.6
2023-01-11T21:12:05.1508067Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2023-01-11T21:12:05.1508468Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2023-01-11T21:12:05.1508751Z   - Magma 2.6.1
2023-01-11T21:12:05.1511858Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2023-01-11T21:12:05.1514065Z 
2023-01-11T21:12:05.3842088Z + cd test
2023-01-11T21:12:05.3842675Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2023-01-11T21:12:06.9816910Z ATen/Parallel:
2023-01-11T21:12:06.9817447Z 	at::get_num_threads() : 16
2023-01-11T21:12:06.9817802Z 	at::get_num_interop_threads() : 16
2023-01-11T21:12:06.9818097Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T21:12:06.9818666Z 	omp_get_max_threads() : 16
2023-01-11T21:12:06.9819328Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T21:12:06.9819702Z 	mkl_get_max_threads() : 16
2023-01-11T21:12:06.9820142Z Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T21:12:06.9820526Z std::thread::hardware_concurrency() : 32
2023-01-11T21:12:06.9820801Z Environment variables:
2023-01-11T21:12:06.9821071Z 	OMP_NUM_THREADS : [not set]
2023-01-11T21:12:06.9821466Z 	MKL_NUM_THREADS : [not set]
2023-01-11T21:12:06.9821736Z ATen parallel backend: OpenMP
2023-01-11T21:12:06.9821913Z 
2023-01-11T21:12:07.2103512Z + [[ distributed == *backward* ]]
2023-01-11T21:12:07.2103872Z + [[ distributed == *xla* ]]
2023-01-11T21:12:07.2104165Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2023-01-11T21:12:07.2104705Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2023-01-11T21:12:07.2105071Z + [[ distributed == distributed ]]
2023-01-11T21:12:07.2105336Z + install_filelock
2023-01-11T21:12:07.2105572Z + pip_install filelock
2023-01-11T21:12:07.2105920Z + pip install --progress-bar off filelock
2023-01-11T21:12:07.7185280Z Collecting filelock
2023-01-11T21:12:07.7360543Z   Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB)
2023-01-11T21:12:08.6456772Z Installing collected packages: filelock
2023-01-11T21:12:08.6820934Z Successfully installed filelock-3.9.0
2023-01-11T21:12:08.7501037Z + install_triton
2023-01-11T21:12:08.7501333Z + local commit
2023-01-11T21:12:08.7501613Z + [[ distributed == *rocm* ]]
2023-01-11T21:12:08.7505761Z ++ get_pinned_commit triton
2023-01-11T21:12:08.7506069Z ++ cat .github/ci_commit_pins/triton.txt
2023-01-11T21:12:08.7520638Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:12:08.7521332Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:12:08.7522037Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:12:09.2178380Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:12:09.2183551Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-d9mojstt
2023-01-11T21:12:09.2204481Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-d9mojstt
2023-01-11T21:12:09.9801980Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2023-01-11T21:12:09.9823024Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:12:10.3992523Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:12:10.7551659Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:12:10.7553793Z   Running command git submodule update --init --recursive -q
2023-01-11T21:12:11.3714483Z   Preparing metadata (setup.py) ... [?25l- done
2023-01-11T21:12:11.6007764Z [?25hCollecting cmake
2023-01-11T21:12:11.6216568Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2023-01-11T21:12:11.9978690Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.9.0)
2023-01-11T21:12:11.9982206Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (2.0.0a0+git8419ddd)
2023-01-11T21:12:12.0243308Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3)
2023-01-11T21:12:12.0248071Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1)
2023-01-11T21:12:12.0252965Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0)
2023-01-11T21:12:12.0465874Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2023-01-11T21:12:12.0537461Z Building wheels for collected packages: triton
2023-01-11T21:13:04.8522464Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / done
2023-01-11T21:13:04.9006287Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=a6e63f0c474c38232aa06213cd5ddd20005b637ac9487d0dbac1cd3076de26f4
2023-01-11T21:13:04.9007421Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295
2023-01-11T21:13:04.9027740Z Successfully built triton
2023-01-11T21:13:05.8070394Z Installing collected packages: cmake, triton
2023-01-11T21:13:07.7900818Z Successfully installed cmake-3.25.0 triton-2.0.0
2023-01-11T21:13:07.8971778Z + pip_install --user jinja2
2023-01-11T21:13:07.8972251Z + pip install --progress-bar off --user jinja2
2023-01-11T21:13:09.0167679Z Collecting jinja2
2023-01-11T21:13:09.0370444Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2023-01-11T21:13:09.3322723Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1)
2023-01-11T21:13:10.2434876Z Installing collected packages: jinja2
2023-01-11T21:13:10.3495196Z Successfully installed jinja2-3.1.2
2023-01-11T21:13:10.4154013Z + test_distributed
2023-01-11T21:13:10.4154514Z + echo 'Testing distributed python tests'
2023-01-11T21:13:10.4154812Z Testing distributed python tests
2023-01-11T21:13:10.4155255Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose
2023-01-11T21:13:12.6337711Z Ignoring disabled issues:  ['91003']
2023-01-11T21:13:12.6725921Z /var/lib/jenkins/workspace/test/run_test.py:1169: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2023-01-11T21:13:12.6726495Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2023-01-11T21:13:12.6731129Z Found test time stats from artifacts
2023-01-11T21:13:12.6749549Z Selected tests:
2023-01-11T21:13:12.6749868Z  distributed/algorithms/quantization/test_quantization
2023-01-11T21:13:12.6750222Z  distributed/test_distributed_spawn
2023-01-11T21:13:12.6750518Z  distributed/rpc/test_tensorpipe_agent
2023-01-11T21:13:12.6750848Z  distributed/pipeline/sync/test_transparency
2023-01-11T21:13:12.6751173Z  distributed/pipeline/sync/test_pipe
2023-01-11T21:13:12.6751475Z  distributed/pipeline/sync/test_inplace
2023-01-11T21:13:12.6751789Z  distributed/pipeline/sync/test_copy
2023-01-11T21:13:12.6754166Z  distributed/pipeline/sync/test_balance
2023-01-11T21:13:12.6754555Z  distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T21:13:12.6754935Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T21:13:12.6755288Z  distributed/optim/test_named_optimizer
2023-01-11T21:13:12.6755566Z  distributed/elastic/timer/api_test
2023-01-11T21:13:12.6755868Z  distributed/_shard/test_sharder
2023-01-11T21:13:12.6756159Z  distributed/test_nccl
2023-01-11T21:13:12.6756448Z  distributed/checkpoint/test_traverse
2023-01-11T21:13:12.6756757Z  distributed/_composable/test_contract
2023-01-11T21:13:12.6757742Z  distributed/checkpoint/test_dedup_tensors
2023-01-11T21:13:12.6758043Z  distributed/checkpoint/test_planner
2023-01-11T21:13:12.6758369Z  distributed/fsdp/test_checkpoint_wrapper
2023-01-11T21:13:12.6758755Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T21:13:12.6759143Z  distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T21:13:12.6759826Z  distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T21:13:12.6760186Z  distributed/checkpoint/test_fsdp_model_state
2023-01-11T21:13:12.6760522Z  distributed/_shard/sharded_tensor/ops/test_chunk
2023-01-11T21:13:12.6760874Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T21:13:12.6761417Z  distributed/_tensor/test_view_ops
2023-01-11T21:13:12.6761814Z  distributed/fsdp/test_distributed_checkpoint
2023-01-11T21:13:12.6762133Z  distributed/elastic/timer/local_timer_example
2023-01-11T21:13:12.6762482Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2023-01-11T21:13:12.6762817Z  distributed/_shard/test_partial_tensor
2023-01-11T21:13:12.6763126Z  distributed/fsdp/test_fsdp_overlap
2023-01-11T21:13:12.6763434Z  distributed/tensor/parallel/test_tp_examples
2023-01-11T21:13:12.6763753Z  distributed/fsdp/test_fsdp_memory
2023-01-11T21:13:12.6764165Z  distributed/test_c10d_object_collectives
2023-01-11T21:13:12.6764498Z  distributed/checkpoint/test_file_system_checkpoint
2023-01-11T21:13:12.6764850Z  distributed/elastic/multiprocessing/api_test
2023-01-11T21:13:12.6765175Z  distributed/_tensor/test_common_rules
2023-01-11T21:13:12.6765474Z  distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T21:13:12.6765818Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T21:13:12.6766157Z  distributed/_composable/test_compose
2023-01-11T21:13:12.6766482Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T21:13:12.6766813Z  distributed/test_c10d_spawn_gloo
2023-01-11T21:13:12.6767114Z  distributed/test_c10d_spawn_nccl
2023-01-11T21:13:12.6767370Z  distributed/test_store
2023-01-11T21:13:12.6767657Z  distributed/_tensor/test_tensor_ops
2023-01-11T21:13:12.6767961Z  distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T21:13:12.6768299Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T21:13:12.6768636Z  distributed/fsdp/test_fsdp_summon_full_params
2023-01-11T21:13:12.6768957Z  distributed/fsdp/test_fsdp_state_dict
2023-01-11T21:13:12.6769250Z  distributed/test_c10d_nccl
2023-01-11T21:13:12.6911557Z Prioritized test from test file changes.
2023-01-11T21:13:12.6911881Z reordering tests for PR:
2023-01-11T21:13:12.6912357Z prioritized: ['distributed/optim/test_named_optimizer']
2023-01-11T21:13:12.6916888Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/rpc/test_tensorpipe_agent', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/elastic/timer/api_test', 'distributed/_shard/test_sharder', 'distributed/test_nccl', 'distributed/checkpoint/test_traverse', 'distributed/_composable/test_contract', 'distributed/checkpoint/test_dedup_tensors', 'distributed/checkpoint/test_planner', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/tensor/parallel/test_view_sharding_dim_change', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/checkpoint/test_fsdp_model_state', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_tensor/test_view_ops', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/elastic/timer/local_timer_example', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/_shard/test_partial_tensor', 'distributed/fsdp/test_fsdp_overlap', 'distributed/tensor/parallel/test_tp_examples', 'distributed/fsdp/test_fsdp_memory', 'distributed/test_c10d_object_collectives', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/elastic/multiprocessing/api_test', 'distributed/_tensor/test_common_rules', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_composable/test_compose', 'distributed/checkpoint/test_file_system_checkpoint_cpu', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_spawn_nccl', 'distributed/test_store', 'distributed/_tensor/test_tensor_ops', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/test_c10d_nccl']
2023-01-11T21:13:12.6920375Z 
2023-01-11T21:13:12.6920930Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2023-01-11T21:13:12.7123922Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2023-01-11T21:13:12.7299452Z parallel (file granularity) tests:
2023-01-11T21:13:12.7299743Z  
2023-01-11T21:13:12.7299980Z serial (file granularity) tests:
2023-01-11T21:13:12.7300294Z  distributed/optim/test_named_optimizer
2023-01-11T21:13:12.7300626Z  distributed/algorithms/quantization/test_quantization
2023-01-11T21:13:12.7300968Z  distributed/test_distributed_spawn
2023-01-11T21:13:12.7301306Z  distributed/rpc/test_tensorpipe_agent
2023-01-11T21:13:12.7301636Z  distributed/pipeline/sync/test_transparency
2023-01-11T21:13:12.7301957Z  distributed/pipeline/sync/test_pipe
2023-01-11T21:13:12.7302246Z  distributed/pipeline/sync/test_inplace
2023-01-11T21:13:12.7302582Z  distributed/pipeline/sync/test_copy
2023-01-11T21:13:12.7302873Z  distributed/pipeline/sync/test_balance
2023-01-11T21:13:12.7303185Z  distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T21:13:12.7303558Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T21:13:12.7303885Z  distributed/elastic/timer/api_test
2023-01-11T21:13:12.7304186Z  distributed/_shard/test_sharder
2023-01-11T21:13:12.7304441Z  distributed/test_nccl
2023-01-11T21:13:12.7304762Z  distributed/checkpoint/test_traverse
2023-01-11T21:13:12.7305056Z  distributed/_composable/test_contract
2023-01-11T21:13:12.7305375Z  distributed/checkpoint/test_dedup_tensors
2023-01-11T21:13:12.7305689Z  distributed/checkpoint/test_planner
2023-01-11T21:13:12.7305958Z  distributed/fsdp/test_checkpoint_wrapper
2023-01-11T21:13:12.7306320Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T21:13:12.7306696Z  distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T21:13:12.7307029Z  distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T21:13:12.7307364Z  distributed/checkpoint/test_fsdp_model_state
2023-01-11T21:13:12.7307708Z  distributed/_shard/sharded_tensor/ops/test_chunk
2023-01-11T21:13:12.7308060Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T21:13:12.7308398Z  distributed/_tensor/test_view_ops
2023-01-11T21:13:12.7308714Z  distributed/fsdp/test_distributed_checkpoint
2023-01-11T21:13:12.7309039Z  distributed/elastic/timer/local_timer_example
2023-01-11T21:13:12.7309390Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2023-01-11T21:13:12.7309726Z  distributed/_shard/test_partial_tensor
2023-01-11T21:13:12.7310036Z  distributed/fsdp/test_fsdp_overlap
2023-01-11T21:13:12.7310335Z  distributed/tensor/parallel/test_tp_examples
2023-01-11T21:13:12.7310662Z  distributed/fsdp/test_fsdp_memory
2023-01-11T21:13:12.7310967Z  distributed/test_c10d_object_collectives
2023-01-11T21:13:12.7311287Z  distributed/checkpoint/test_file_system_checkpoint
2023-01-11T21:13:12.7311637Z  distributed/elastic/multiprocessing/api_test
2023-01-11T21:13:12.7311970Z  distributed/_tensor/test_common_rules
2023-01-11T21:13:12.7312268Z  distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T21:13:12.7312611Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T21:13:12.7312941Z  distributed/_composable/test_compose
2023-01-11T21:13:12.7313270Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T21:13:12.7313596Z  distributed/test_c10d_spawn_gloo
2023-01-11T21:13:12.7313888Z  distributed/test_c10d_spawn_nccl
2023-01-11T21:13:12.7314148Z  distributed/test_store
2023-01-11T21:13:12.7314432Z  distributed/_tensor/test_tensor_ops
2023-01-11T21:13:12.7314738Z  distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T21:13:12.7315068Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T21:13:12.7315508Z  distributed/fsdp/test_fsdp_summon_full_params
2023-01-11T21:13:12.7315826Z  distributed/fsdp/test_fsdp_state_dict
2023-01-11T21:13:12.7316116Z  distributed/test_c10d_nccl
2023-01-11T21:13:14.9196003Z Ignoring disabled issues:  ['91003']
2023-01-11T21:13:14.9335913Z Ignoring disabled issues:  ['91003']
2023-01-11T21:13:15.3457147Z Running distributed/optim/test_named_optimizer ... [2023-01-11 21:13:15.345206]
2023-01-11T21:13:15.3460752Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:13:15.345722]
2023-01-11T21:13:17.4744404Z 
2023-01-11T21:13:17.4745314Z Expand the folded group to see the log file of distributed/optim/test_named_optimizer
2023-01-11T21:13:17.4752324Z ##[group]PRINTING LOG FILE of distributed/optim/test_named_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_named_optimizer_xh7p52nc)
2023-01-11T21:13:17.4752735Z 
2023-01-11T21:13:17.4753061Z ##[endgroup]
2023-01-11T21:13:17.4753813Z FINISHED PRINTING LOG FILE of distributed/optim/test_named_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_named_optimizer_xh7p52nc)
2023-01-11T21:13:17.4754184Z 
2023-01-11T21:13:17.4754503Z Running distributed/algorithms/quantization/test_quantization ... [2023-01-11 21:13:17.474491]
2023-01-11T21:13:17.4757542Z /usr/bin/mpiexec
2023-01-11T21:13:17.4758661Z MPI not available -- MPI backend tests will be skipped
2023-01-11T21:13:17.4760257Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2023-01-11T21:13:17.4760683Z Shard 2: test should be run in 1
2023-01-11T21:13:17.4763455Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2023-01-11T21:13:17.4768436Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:13:17.476490]
2023-01-11T21:13:59.2510120Z 
2023-01-11T21:13:59.2511070Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:13:59.2512232Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ebiq74x5)
2023-01-11T21:13:59.2513360Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T21:13:59.2514245Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:13:59.2514606Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:13:59.2514974Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:13:59.2515334Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:13:59.2515727Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:13:59.2516071Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:13:59.2516421Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:13:59.2517136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2517584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2518148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2518619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2518846Z 
2023-01-11T21:13:59.2523067Z Running tests...
2023-01-11T21:13:59.2523802Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2524457Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:13:59.2525027Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:13:59.2525325Z 
2023-01-11T21:13:59.2525591Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2525901Z Ran 1 test in 0.001s
2023-01-11T21:13:59.2526064Z 
2023-01-11T21:13:59.2526290Z OK (skipped=1)
2023-01-11T21:13:59.2526463Z 
2023-01-11T21:13:59.2526590Z Generating XML reports...
2023-01-11T21:13:59.2527246Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211321.xml
2023-01-11T21:13:59.2528012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2528479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2529071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2529516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2529756Z 
2023-01-11T21:13:59.2529871Z Running tests...
2023-01-11T21:13:59.2530267Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2530859Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:13:59.2531431Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:13:59.2531749Z 
2023-01-11T21:13:59.2532016Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2532341Z Ran 1 test in 0.001s
2023-01-11T21:13:59.2532511Z 
2023-01-11T21:13:59.2532604Z OK (skipped=1)
2023-01-11T21:13:59.2532757Z 
2023-01-11T21:13:59.2532883Z Generating XML reports...
2023-01-11T21:13:59.2533541Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211323.xml
2023-01-11T21:13:59.2534310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2534732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2535313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2535782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2536012Z 
2023-01-11T21:13:59.2536121Z Running tests...
2023-01-11T21:13:59.2536501Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2537100Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:13:59.2537647Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:13:59.2538100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1029
2023-01-11T21:13:59.2538541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1030
2023-01-11T21:13:59.2539143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2539594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2540149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2540613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2541187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2541689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2542260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2542723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2543155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:13:59.2543666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:13:59.2544154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:13:59.2544645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:13:59.2545304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2545825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:13:59.2546469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2546989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:13:59.2547618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2548307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2548699Z ok (6.649s)
2023-01-11T21:13:59.2548847Z 
2023-01-11T21:13:59.2549119Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2549430Z Ran 1 test in 6.649s
2023-01-11T21:13:59.2549591Z 
2023-01-11T21:13:59.2549695Z OK
2023-01-11T21:13:59.2549822Z 
2023-01-11T21:13:59.2549949Z Generating XML reports...
2023-01-11T21:13:59.2550591Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211325.xml
2023-01-11T21:13:59.2551347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2551798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2552378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2552827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2553055Z 
2023-01-11T21:13:59.2553164Z Running tests...
2023-01-11T21:13:59.2553567Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2554169Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:13:59.2554700Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:13:59.2555163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1148
2023-01-11T21:13:59.2555601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1149
2023-01-11T21:13:59.2556186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2556687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2557265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2557734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2558309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2558806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2559978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2560522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2560942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:13:59.2561416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:13:59.2562003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:13:59.2562515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:13:59.2563172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2563702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:13:59.2564346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2564875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:13:59.2565503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2566180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2566576Z ok (6.628s)
2023-01-11T21:13:59.2566722Z 
2023-01-11T21:13:59.2566987Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2567296Z Ran 1 test in 6.628s
2023-01-11T21:13:59.2567457Z 
2023-01-11T21:13:59.2567550Z OK
2023-01-11T21:13:59.2567683Z 
2023-01-11T21:13:59.2567809Z Generating XML reports...
2023-01-11T21:13:59.2568459Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211334.xml
2023-01-11T21:13:59.2569210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2569654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2570222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2570673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2570901Z 
2023-01-11T21:13:59.2571010Z Running tests...
2023-01-11T21:13:59.2571413Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2571988Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:13:59.2572536Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:13:59.2573015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1267
2023-01-11T21:13:59.2573457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1268
2023-01-11T21:13:59.2574041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2574488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2575061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2575505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2576078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2576566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2577236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2577680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2578116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:13:59.2578607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:13:59.2579162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:13:59.2579636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:13:59.2580295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2580818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:13:59.2581450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2581980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:13:59.2582620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2583292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2583668Z ok (6.656s)
2023-01-11T21:13:59.2583815Z 
2023-01-11T21:13:59.2584081Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2584404Z Ran 1 test in 6.656s
2023-01-11T21:13:59.2584564Z 
2023-01-11T21:13:59.2584639Z OK
2023-01-11T21:13:59.2584772Z 
2023-01-11T21:13:59.2584894Z Generating XML reports...
2023-01-11T21:13:59.2585552Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211343.xml
2023-01-11T21:13:59.2586312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2586742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2587309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2587782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2588011Z 
2023-01-11T21:13:59.2588120Z Running tests...
2023-01-11T21:13:59.2588505Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2589103Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:13:59.2589647Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:13:59.2590109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1386
2023-01-11T21:13:59.2590550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1387
2023-01-11T21:13:59.2591151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2591592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2592149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2592610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2593187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:13:59.2593610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:13:59.2594170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:13:59.2594698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:13:59.2595135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:13:59.2595585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:13:59.2596065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:13:59.2596600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:13:59.2597268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2597783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:13:59.2598425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:13:59.2598953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:13:59.2600225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2600937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:13:59.2601329Z ok (6.629s)
2023-01-11T21:13:59.2601482Z 
2023-01-11T21:13:59.2601750Z ----------------------------------------------------------------------
2023-01-11T21:13:59.2602057Z Ran 1 test in 6.629s
2023-01-11T21:13:59.2602215Z 
2023-01-11T21:13:59.2602308Z OK
2023-01-11T21:13:59.2602441Z 
2023-01-11T21:13:59.2602566Z Generating XML reports...
2023-01-11T21:13:59.2603209Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211351.xml
2023-01-11T21:13:59.2603613Z 
2023-01-11T21:13:59.2604050Z ##[endgroup]
2023-01-11T21:13:59.2604768Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ebiq74x5)
2023-01-11T21:13:59.2605202Z 
2023-01-11T21:13:59.2605416Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2023-01-11T21:13:59.2606181Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:13:59.252585]
2023-01-11T21:14:40.9195366Z 
2023-01-11T21:14:40.9195919Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:14:40.9200481Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_yqq9gz3o)
2023-01-11T21:14:40.9201670Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T21:14:40.9202519Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:14:40.9202874Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:14:40.9203236Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:14:40.9203638Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:14:40.9204007Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:14:40.9204613Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:14:40.9204968Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:14:40.9205667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9206345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9206943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9207541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9207790Z 
2023-01-11T21:14:40.9207913Z Running tests...
2023-01-11T21:14:40.9208306Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9208913Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:14:40.9209710Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:14:40.9210013Z 
2023-01-11T21:14:40.9210289Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9210589Z Ran 1 test in 0.001s
2023-01-11T21:14:40.9210746Z 
2023-01-11T21:14:40.9210855Z OK (skipped=1)
2023-01-11T21:14:40.9211017Z 
2023-01-11T21:14:40.9211135Z Generating XML reports...
2023-01-11T21:14:40.9211783Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211402.xml
2023-01-11T21:14:40.9212545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9212988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9213560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9214002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9214235Z 
2023-01-11T21:14:40.9214345Z Running tests...
2023-01-11T21:14:40.9214961Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9216095Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:14:40.9217159Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:14:40.9217681Z 
2023-01-11T21:14:40.9218223Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9218926Z Ran 1 test in 0.001s
2023-01-11T21:14:40.9219264Z 
2023-01-11T21:14:40.9219450Z OK (skipped=1)
2023-01-11T21:14:40.9219640Z 
2023-01-11T21:14:40.9219791Z Generating XML reports...
2023-01-11T21:14:40.9221082Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211405.xml
2023-01-11T21:14:40.9222433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9223350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9224603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9225548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9226016Z 
2023-01-11T21:14:40.9226233Z Running tests...
2023-01-11T21:14:40.9226982Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9228311Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:14:40.9229282Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:14:40.9229749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1604
2023-01-11T21:14:40.9230315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1605
2023-01-11T21:14:40.9230910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9231358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9231928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9232392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9233010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9233462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9234031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9234490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9234907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:40.9235373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:40.9235855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:14:40.9236329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:14:40.9236989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9237521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:14:40.9238165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9238674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:14:40.9239314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9240685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9241078Z ok (6.634s)
2023-01-11T21:14:40.9241208Z 
2023-01-11T21:14:40.9241474Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9241802Z Ran 1 test in 6.634s
2023-01-11T21:14:40.9241965Z 
2023-01-11T21:14:40.9242057Z OK
2023-01-11T21:14:40.9242190Z 
2023-01-11T21:14:40.9242296Z Generating XML reports...
2023-01-11T21:14:40.9242956Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211407.xml
2023-01-11T21:14:40.9243707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9244155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9244710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9245172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9245399Z 
2023-01-11T21:14:40.9245507Z Running tests...
2023-01-11T21:14:40.9245889Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9246487Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:14:40.9247022Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:14:40.9247486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1723
2023-01-11T21:14:40.9247902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1724
2023-01-11T21:14:40.9248639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9249083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9249653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9250100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9250741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9251194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9251745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9252204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9252637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:40.9253099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:40.9253568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:14:40.9254055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:14:40.9254713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9255382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9255906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:14:40.9256387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:14:40.9257027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9257686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9258071Z ok (6.621s)
2023-01-11T21:14:40.9258219Z 
2023-01-11T21:14:40.9258484Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9258806Z Ran 1 test in 6.621s
2023-01-11T21:14:40.9258949Z 
2023-01-11T21:14:40.9259046Z OK
2023-01-11T21:14:40.9259178Z 
2023-01-11T21:14:40.9259299Z Generating XML reports...
2023-01-11T21:14:40.9260017Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211415.xml
2023-01-11T21:14:40.9260747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9261196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9261763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9262220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9262444Z 
2023-01-11T21:14:40.9262536Z Running tests...
2023-01-11T21:14:40.9262933Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9263533Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:14:40.9264069Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:14:40.9264542Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1842
2023-01-11T21:14:40.9264983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1843
2023-01-11T21:14:40.9265656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9266080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9266647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9267107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9267734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9268165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9268733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9269194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9269608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:40.9270079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:40.9270560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:14:40.9271049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:14:40.9271682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9272209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:14:40.9272848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9273373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:14:40.9274001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9274674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9275063Z ok (6.637s)
2023-01-11T21:14:40.9275210Z 
2023-01-11T21:14:40.9275459Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9275781Z Ran 1 test in 6.637s
2023-01-11T21:14:40.9275940Z 
2023-01-11T21:14:40.9276033Z OK
2023-01-11T21:14:40.9276166Z 
2023-01-11T21:14:40.9276294Z Generating XML reports...
2023-01-11T21:14:40.9276937Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211424.xml
2023-01-11T21:14:40.9277682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9278129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9278683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9279151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9280159Z 
2023-01-11T21:14:40.9280314Z Running tests...
2023-01-11T21:14:40.9280730Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9281405Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:14:40.9281961Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:14:40.9282435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1961
2023-01-11T21:14:40.9282857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1962
2023-01-11T21:14:40.9283459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9284027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9284602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9285051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9285623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:40.9286137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:40.9286728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:40.9287176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:40.9287604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:40.9288096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:14:40.9288557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:40.9289031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:14:40.9289693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9290225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:14:40.9290848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:14:40.9291370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:14:40.9292002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9292681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:14:40.9293050Z ok (6.533s)
2023-01-11T21:14:40.9293195Z 
2023-01-11T21:14:40.9293459Z ----------------------------------------------------------------------
2023-01-11T21:14:40.9293781Z Ran 1 test in 6.533s
2023-01-11T21:14:40.9293957Z 
2023-01-11T21:14:40.9294032Z OK
2023-01-11T21:14:40.9294165Z 
2023-01-11T21:14:40.9294287Z Generating XML reports...
2023-01-11T21:14:40.9294948Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211433.xml
2023-01-11T21:14:40.9295346Z 
2023-01-11T21:14:40.9295677Z ##[endgroup]
2023-01-11T21:14:40.9296382Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_yqq9gz3o)
2023-01-11T21:14:40.9296820Z 
2023-01-11T21:14:40.9296947Z Shard 2: gloo should be run in 1
2023-01-11T21:14:40.9297224Z Shard 2: ucc should be run in 1
2023-01-11T21:14:40.9297628Z Running distributed/test_distributed_spawn ... [2023-01-11 21:14:40.920849]
2023-01-11T21:14:40.9297937Z /usr/bin/mpiexec
2023-01-11T21:14:40.9298295Z MPI not available -- MPI backend tests will be skipped
2023-01-11T21:14:40.9298825Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3}
2023-01-11T21:14:40.9299214Z Shard 2: test should be run in 1
2023-01-11T21:14:40.9299577Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2023-01-11T21:14:40.9300310Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:40.922561]
2023-01-11T21:40:54.3350037Z 
2023-01-11T21:40:54.3352605Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T21:40:54.3353536Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_wk42wko9)
2023-01-11T21:40:54.3357026Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:40:54.3394137Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T21:40:54.3429480Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3431111Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3431545Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3431960Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3432392Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3432984Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3433478Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3433976Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3434467Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3435080Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3435656Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3436178Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3436703Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3437211Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3437700Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3438170Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3438645Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3439074Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3440013Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3440454Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3440949Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3441404Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3441796Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3442190Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3442628Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3443043Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3443434Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3443866Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3444283Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3444630Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3445022Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3445436Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3445800Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3446207Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3446611Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3447040Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3447410Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3447825Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3448249Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3448631Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3449032Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3449460Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3449902Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3450335Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3450871Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3451298Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3451717Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3452153Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3452551Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3453032Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3453479Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3453891Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3454299Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3454706Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3455152Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3455539Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3455942Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3456346Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3456753Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3457155Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3457526Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3457912Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3458282Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3458637Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3459016Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3459423Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3459807Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3460198Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3460573Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3460950Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3461319Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3461712Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3462101Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3462488Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3462870Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3463241Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3463601Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3463986Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3464378Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3464774Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3465148Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3465533Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3465937Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3466347Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3466779Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3467216Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3467715Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3468168Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3468611Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3469051Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3469470Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3469949Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3470391Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3470820Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3471274Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3471740Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3472192Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3472619Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3473038Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3473426Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3473784Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3474166Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3474512Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3475021Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3475881Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3476592Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3477288Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3478008Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3478750Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3479976Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3480753Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3481601Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3482472Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3483323Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3484137Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3484960Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3485749Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3486558Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3487381Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3488170Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3488960Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3489681Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3490444Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3491212Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3491953Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3492716Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3493612Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3494640Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3495831Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3496764Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3497744Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3498594Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3499444Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3500434Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3501359Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3502256Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3503232Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3504140Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3504972Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3505873Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3506723Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3507471Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3508226Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3509075Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3509938Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3510802Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3511689Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3512534Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3513465Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3514377Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3515419Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3516553Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3517831Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3519102Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3520830Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3522127Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3523354Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3524552Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3525777Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3526890Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3527907Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3528808Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3529783Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3530600Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3531408Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3532213Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3533049Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3534125Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3535047Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3535985Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3536810Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3537547Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3538388Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3539289Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3540144Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3541027Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3541880Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3542786Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3543646Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3544504Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3545333Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3546139Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3546934Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3547755Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3548609Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3549496Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3550400Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3551187Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3551874Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3552663Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3553503Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3554333Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3555096Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3555856Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3556586Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3557322Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3558074Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3558814Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3560151Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3560925Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3561669Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3562366Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3563128Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3563934Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3564718Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3565667Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3566355Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3567101Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3567911Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3568716Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3569642Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3570697Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3571525Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3572365Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3573245Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3574121Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3574957Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3575790Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3576620Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3577419Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3578241Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3579047Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3579831Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3580808Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3581874Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3582833Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3583733Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3584649Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3585569Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3586422Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3587271Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3588138Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3589029Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3589927Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3590912Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3591996Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3592940Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3593789Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3594590Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3595393Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3596178Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3596951Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3597712Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3598476Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3599263Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3600560Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3601475Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3602253Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3603013Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3603813Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3604613Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3605346Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3606251Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3607013Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3607769Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3608490Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3609187Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3609942Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3610692Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3611466Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3612235Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3612990Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3613843Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3614558Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3615319Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3616183Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3617061Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3617920Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3618693Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3619499Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3620346Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3621192Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3621997Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3622845Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3623729Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3624568Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3625363Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3626190Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3626998Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3627768Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3628591Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3629524Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3630438Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.3631818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3632740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3633920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3634882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3635346Z 
2023-01-11T21:40:54.3635520Z Running tests...
2023-01-11T21:40:54.3636324Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3637565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3638726Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3640275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2114
2023-01-11T21:40:54.3641164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2115
2023-01-11T21:40:54.3642597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3643512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3644695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3645654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3646862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3647743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3648947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3649886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3650761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3651734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3653104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3654462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3655469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3656397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3657548Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3659284Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3660605Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3662309Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3663665Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3665394Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3666684Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3668467Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3669797Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3671513Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3672873Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3674806Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3676170Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3678005Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3679334Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:40:54.3681439Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:40:54.3682409Z ok (6.949s)
2023-01-11T21:40:54.3682705Z 
2023-01-11T21:40:54.3683246Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3683866Z Ran 1 test in 6.950s
2023-01-11T21:40:54.3684177Z 
2023-01-11T21:40:54.3684344Z OK
2023-01-11T21:40:54.3684605Z 
2023-01-11T21:40:54.3684843Z Generating XML reports...
2023-01-11T21:40:54.3686072Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211445.xml
2023-01-11T21:40:54.3687567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3688482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3689652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3690578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3691034Z 
2023-01-11T21:40:54.3691263Z Running tests...
2023-01-11T21:40:54.3692073Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3693142Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3694207Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T21:40:54.3694835Z 
2023-01-11T21:40:54.3695364Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3695995Z Ran 1 test in 0.003s
2023-01-11T21:40:54.3696313Z 
2023-01-11T21:40:54.3696495Z OK (skipped=1)
2023-01-11T21:40:54.3696783Z 
2023-01-11T21:40:54.3697012Z Generating XML reports...
2023-01-11T21:40:54.3698222Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211454.xml
2023-01-11T21:40:54.3699682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3700573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3701751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3702693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3703156Z 
2023-01-11T21:40:54.3703338Z Running tests...
2023-01-11T21:40:54.3704169Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3705254Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3706265Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3707271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2262
2023-01-11T21:40:54.3708220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2263
2023-01-11T21:40:54.3709698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3710624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3711824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3712779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3714348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3715248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3716439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3717374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3718293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3719278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3720983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3722441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3723495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3724461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3725166Z ok (4.321s)
2023-01-11T21:40:54.3725446Z 
2023-01-11T21:40:54.3726004Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3726628Z Ran 1 test in 4.321s
2023-01-11T21:40:54.3726933Z 
2023-01-11T21:40:54.3727111Z OK
2023-01-11T21:40:54.3727373Z 
2023-01-11T21:40:54.3727609Z Generating XML reports...
2023-01-11T21:40:54.3728784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211457.xml
2023-01-11T21:40:54.3730252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3731151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3732342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3733269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3733729Z 
2023-01-11T21:40:54.3733940Z Running tests...
2023-01-11T21:40:54.3734762Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3735801Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3736883Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3739044Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s)
2023-01-11T21:40:54.3740139Z 
2023-01-11T21:40:54.3740700Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3741356Z Ran 1 test in 1.616s
2023-01-11T21:40:54.3741684Z 
2023-01-11T21:40:54.3741871Z OK (skipped=1)
2023-01-11T21:40:54.3742181Z 
2023-01-11T21:40:54.3742423Z Generating XML reports...
2023-01-11T21:40:54.3743643Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211503.xml
2023-01-11T21:40:54.3745113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3746241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3747438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3748371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3748834Z 
2023-01-11T21:40:54.3749013Z Running tests...
2023-01-11T21:40:54.3750021Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3751126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3752216Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T21:40:54.3752876Z 
2023-01-11T21:40:54.3753413Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3754055Z Ran 1 test in 0.002s
2023-01-11T21:40:54.3754357Z 
2023-01-11T21:40:54.3754563Z OK (skipped=1)
2023-01-11T21:40:54.3754859Z 
2023-01-11T21:40:54.3755077Z Generating XML reports...
2023-01-11T21:40:54.3756332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211508.xml
2023-01-11T21:40:54.3757797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3758716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3760231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3761212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3761666Z 
2023-01-11T21:40:54.3761874Z Running tests...
2023-01-11T21:40:54.3762668Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3763769Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3764907Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T21:40:54.3765617Z 
2023-01-11T21:40:54.3766175Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3766800Z Ran 1 test in 0.002s
2023-01-11T21:40:54.3767109Z 
2023-01-11T21:40:54.3767327Z OK (skipped=1)
2023-01-11T21:40:54.3767631Z 
2023-01-11T21:40:54.3767882Z Generating XML reports...
2023-01-11T21:40:54.3769124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211510.xml
2023-01-11T21:40:54.3770570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3771490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3772694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3773614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3774070Z 
2023-01-11T21:40:54.3774285Z Running tests...
2023-01-11T21:40:54.3775080Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3776134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3777235Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3778304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2467
2023-01-11T21:40:54.3779197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2468
2023-01-11T21:40:54.3780404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3781500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3782691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3783649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3784824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3785892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3787096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3788025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3788912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3789928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3791274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3792674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3793725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3794687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3795647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3796597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3797588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3798587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3799840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3800842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3801827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3802830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3803802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3804774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3805748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3806722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3807429Z ok (7.968s)
2023-01-11T21:40:54.3807706Z 
2023-01-11T21:40:54.3808253Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3808883Z Ran 1 test in 7.968s
2023-01-11T21:40:54.3809199Z 
2023-01-11T21:40:54.3809348Z OK
2023-01-11T21:40:54.3809601Z 
2023-01-11T21:40:54.3809837Z Generating XML reports...
2023-01-11T21:40:54.3811082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211512.xml
2023-01-11T21:40:54.3812583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3813556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3814745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3815661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3816341Z 
2023-01-11T21:40:54.3816528Z Running tests...
2023-01-11T21:40:54.3817351Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3818427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3819591Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3820656Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2586
2023-01-11T21:40:54.3821712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2587
2023-01-11T21:40:54.3822999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3823885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3825089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3826052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3827254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3828131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3829344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3830293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3831172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3832174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3833536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3834966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3836009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3836950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3837916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3838897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3840232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3841223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3841914Z ok (6.918s)
2023-01-11T21:40:54.3842206Z 
2023-01-11T21:40:54.3842739Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3843404Z Ran 1 test in 6.918s
2023-01-11T21:40:54.3843707Z 
2023-01-11T21:40:54.3843875Z OK
2023-01-11T21:40:54.3844129Z 
2023-01-11T21:40:54.3844353Z Generating XML reports...
2023-01-11T21:40:54.3845581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211523.xml
2023-01-11T21:40:54.3847068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3847979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3849165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3850121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3850571Z 
2023-01-11T21:40:54.3850772Z Running tests...
2023-01-11T21:40:54.3851559Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3852851Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3854043Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3855165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2705
2023-01-11T21:40:54.3856045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2706
2023-01-11T21:40:54.3857460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3858386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3859582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3860505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3861712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3862608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3863791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3864712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3865620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3866601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3867944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3869476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3870556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3871537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3872506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3873513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3874522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3875505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3876473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3877449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3878419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3879830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3880550Z ok (7.030s)
2023-01-11T21:40:54.3880837Z 
2023-01-11T21:40:54.3881408Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3882043Z Ran 1 test in 7.031s
2023-01-11T21:40:54.3882324Z 
2023-01-11T21:40:54.3882485Z OK
2023-01-11T21:40:54.3882727Z 
2023-01-11T21:40:54.3882962Z Generating XML reports...
2023-01-11T21:40:54.3884237Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211532.xml
2023-01-11T21:40:54.3885664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3886572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3887754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3888909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3889340Z 
2023-01-11T21:40:54.3889539Z Running tests...
2023-01-11T21:40:54.3890345Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3891405Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3892735Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3893893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2824
2023-01-11T21:40:54.3894783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2825
2023-01-11T21:40:54.3896056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3896952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3898150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3899123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3900294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3901212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3902407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3903346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3904244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3905258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3906645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3908096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3909136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3910118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3911117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3912102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3912821Z ok (7.033s)
2023-01-11T21:40:54.3913109Z 
2023-01-11T21:40:54.3913764Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3914414Z Ran 1 test in 7.033s
2023-01-11T21:40:54.3914704Z 
2023-01-11T21:40:54.3914864Z OK
2023-01-11T21:40:54.3915125Z 
2023-01-11T21:40:54.3915359Z Generating XML reports...
2023-01-11T21:40:54.3916603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211542.xml
2023-01-11T21:40:54.3918082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3919015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3920518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3921479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3921944Z 
2023-01-11T21:40:54.3922123Z Running tests...
2023-01-11T21:40:54.3922929Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3924010Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3925405Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3926569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2943
2023-01-11T21:40:54.3927480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2944
2023-01-11T21:40:54.3928887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3929792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3930982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3931928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3933131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3934042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3935220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3936168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3937052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3938084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3939450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3940888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3941928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3942896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3943881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3944869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3945818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3946812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3947516Z ok (7.645s)
2023-01-11T21:40:54.3947798Z 
2023-01-11T21:40:54.3948337Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3948960Z Ran 1 test in 7.646s
2023-01-11T21:40:54.3949259Z 
2023-01-11T21:40:54.3949421Z OK
2023-01-11T21:40:54.3949667Z 
2023-01-11T21:40:54.3949902Z Generating XML reports...
2023-01-11T21:40:54.3951148Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211551.xml
2023-01-11T21:40:54.3952604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3953514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3954665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3955622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3956088Z 
2023-01-11T21:40:54.3956291Z Running tests...
2023-01-11T21:40:54.3957201Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3958252Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3959662Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3960976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3062
2023-01-11T21:40:54.3961863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3063
2023-01-11T21:40:54.3963134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3964061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3965399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3966331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3967539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3968438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3969655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3970559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3971473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.3972472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.3973871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3975297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.3976366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.3977301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.3978272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3979275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3980231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3981126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.3981805Z ok (7.444s)
2023-01-11T21:40:54.3982109Z 
2023-01-11T21:40:54.3982663Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3983294Z Ran 1 test in 7.444s
2023-01-11T21:40:54.3983574Z 
2023-01-11T21:40:54.3983742Z OK
2023-01-11T21:40:54.3983997Z 
2023-01-11T21:40:54.3984237Z Generating XML reports...
2023-01-11T21:40:54.3985469Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211602.xml
2023-01-11T21:40:54.3986938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3987864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3989049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.3990028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.3990510Z 
2023-01-11T21:40:54.3990698Z Running tests...
2023-01-11T21:40:54.3991524Z ----------------------------------------------------------------------
2023-01-11T21:40:54.3992622Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.3993792Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.3994952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3181
2023-01-11T21:40:54.3996006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3182
2023-01-11T21:40:54.3997274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.3998164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.3999348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4000770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4001993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4002871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4004023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4004974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4005841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4006843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4008194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4009637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4010659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4011595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4012573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4013661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4014611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4015576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4016275Z ok (6.939s)
2023-01-11T21:40:54.4016553Z 
2023-01-11T21:40:54.4017096Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4017728Z Ran 1 test in 6.939s
2023-01-11T21:40:54.4018032Z 
2023-01-11T21:40:54.4018204Z OK
2023-01-11T21:40:54.4018472Z 
2023-01-11T21:40:54.4018703Z Generating XML reports...
2023-01-11T21:40:54.4019902Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211612.xml
2023-01-11T21:40:54.4021364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4022288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4023459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4024394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4024865Z 
2023-01-11T21:40:54.4025068Z Running tests...
2023-01-11T21:40:54.4025871Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4026940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4028059Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4030259Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2023-01-11T21:40:54.4031536Z 
2023-01-11T21:40:54.4032090Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4032719Z Ran 1 test in 1.613s
2023-01-11T21:40:54.4033041Z 
2023-01-11T21:40:54.4033238Z OK (skipped=1)
2023-01-11T21:40:54.4033531Z 
2023-01-11T21:40:54.4033766Z Generating XML reports...
2023-01-11T21:40:54.4034979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211621.xml
2023-01-11T21:40:54.4036555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4037471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4038629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4039908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4040395Z 
2023-01-11T21:40:54.4040600Z Running tests...
2023-01-11T21:40:54.4041426Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4042517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4043596Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4044698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3334
2023-01-11T21:40:54.4045604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3335
2023-01-11T21:40:54.4046867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4047800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4048980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4049957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4051129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4052046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4053214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4054178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4055077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4056071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4057444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4058835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4059895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4060855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4061544Z ok (4.329s)
2023-01-11T21:40:54.4061803Z 
2023-01-11T21:40:54.4062342Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4062980Z Ran 1 test in 4.329s
2023-01-11T21:40:54.4063297Z 
2023-01-11T21:40:54.4063472Z OK
2023-01-11T21:40:54.4063730Z 
2023-01-11T21:40:54.4063938Z Generating XML reports...
2023-01-11T21:40:54.4065180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211625.xml
2023-01-11T21:40:54.4066652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4067772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4068920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4069880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4070340Z 
2023-01-11T21:40:54.4070565Z Running tests...
2023-01-11T21:40:54.4071549Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4072637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4073735Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4075940Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.624s)
2023-01-11T21:40:54.4077005Z 
2023-01-11T21:40:54.4077561Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4078188Z Ran 1 test in 1.624s
2023-01-11T21:40:54.4078502Z 
2023-01-11T21:40:54.4078703Z OK (skipped=1)
2023-01-11T21:40:54.4078998Z 
2023-01-11T21:40:54.4079229Z Generating XML reports...
2023-01-11T21:40:54.4080854Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211632.xml
2023-01-11T21:40:54.4082350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4083257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4084455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4085406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4085876Z 
2023-01-11T21:40:54.4086074Z Running tests...
2023-01-11T21:40:54.4086885Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4087927Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4088998Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4090024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3473
2023-01-11T21:40:54.4090916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3474
2023-01-11T21:40:54.4092163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4093078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4094276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4095204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4096388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4097297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4098465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4099416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4100300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4101286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4102824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4104278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4105363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4106315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4107178Z ok (6.938s)
2023-01-11T21:40:54.4107487Z 
2023-01-11T21:40:54.4108034Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4108685Z Ran 1 test in 6.938s
2023-01-11T21:40:54.4108996Z 
2023-01-11T21:40:54.4109163Z OK
2023-01-11T21:40:54.4109401Z 
2023-01-11T21:40:54.4109636Z Generating XML reports...
2023-01-11T21:40:54.4110870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211636.xml
2023-01-11T21:40:54.4112341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4113228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4114481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4175975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4176500Z 
2023-01-11T21:40:54.4176734Z Running tests...
2023-01-11T21:40:54.4177559Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4178662Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4179671Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2023-01-11T21:40:54.4180244Z 
2023-01-11T21:40:54.4180798Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4181409Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4181710Z 
2023-01-11T21:40:54.4181909Z OK (skipped=1)
2023-01-11T21:40:54.4182208Z 
2023-01-11T21:40:54.4182432Z Generating XML reports...
2023-01-11T21:40:54.4183657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211646.xml
2023-01-11T21:40:54.4185132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4186061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4187243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4188171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4188619Z 
2023-01-11T21:40:54.4188826Z Running tests...
2023-01-11T21:40:54.4189648Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4190707Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4191633Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.4192570Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4193538Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3621
2023-01-11T21:40:54.4194428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3622
2023-01-11T21:40:54.4195696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4196613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4197774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4198929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4200457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4201368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4202514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4203634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4204588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4205546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4206922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4208347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4209421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4210353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4211049Z ok (7.433s)
2023-01-11T21:40:54.4211339Z 
2023-01-11T21:40:54.4211893Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4212532Z Ran 1 test in 7.433s
2023-01-11T21:40:54.4212822Z 
2023-01-11T21:40:54.4212988Z OK
2023-01-11T21:40:54.4213245Z 
2023-01-11T21:40:54.4213562Z Generating XML reports...
2023-01-11T21:40:54.4214767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211648.xml
2023-01-11T21:40:54.4216208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4217133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4218319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4219260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4219701Z 
2023-01-11T21:40:54.4219903Z Running tests...
2023-01-11T21:40:54.4220720Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4221784Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4222735Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.4223743Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4224716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3740
2023-01-11T21:40:54.4225634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3741
2023-01-11T21:40:54.4226854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4227759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4228942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4229872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4231088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4231980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4233146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4234263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4235168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4236160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4237535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4239050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4240538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4241531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4242495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4243498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4244207Z ok (7.345s)
2023-01-11T21:40:54.4244501Z 
2023-01-11T21:40:54.4245067Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4245688Z Ran 1 test in 7.345s
2023-01-11T21:40:54.4245997Z 
2023-01-11T21:40:54.4246158Z OK
2023-01-11T21:40:54.4246413Z 
2023-01-11T21:40:54.4246650Z Generating XML reports...
2023-01-11T21:40:54.4247880Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211658.xml
2023-01-11T21:40:54.4249357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4250267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4251462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4252402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4252865Z 
2023-01-11T21:40:54.4253066Z Running tests...
2023-01-11T21:40:54.4253873Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4254912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4255967Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.4257171Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4258183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3859
2023-01-11T21:40:54.4259085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3860
2023-01-11T21:40:54.4260356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4261294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4262447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4263420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4264618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4265524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4266732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4267689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4268592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4269594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4271157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4272566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4273579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4274490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4275611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4276612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.4277318Z ok (7.245s)
2023-01-11T21:40:54.4277618Z 
2023-01-11T21:40:54.4278142Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4278816Z Ran 1 test in 7.245s
2023-01-11T21:40:54.4279127Z 
2023-01-11T21:40:54.4279301Z OK
2023-01-11T21:40:54.4279887Z 
2023-01-11T21:40:54.4280107Z Generating XML reports...
2023-01-11T21:40:54.4281351Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211708.xml
2023-01-11T21:40:54.4282789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4283682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4284858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4285804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4286256Z 
2023-01-11T21:40:54.4286487Z Running tests...
2023-01-11T21:40:54.4287283Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4288387Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4289334Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:40:54.4290307Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4291245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3978
2023-01-11T21:40:54.4292160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3979
2023-01-11T21:40:54.4293409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4294295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4295401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4296325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4297386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4298166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4299230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4300120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4300971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4301877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4303145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4304510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4305701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4306571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4307202Z ok (7.333s)
2023-01-11T21:40:54.4307470Z 
2023-01-11T21:40:54.4307971Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4308539Z Ran 1 test in 7.333s
2023-01-11T21:40:54.4308824Z 
2023-01-11T21:40:54.4308975Z OK
2023-01-11T21:40:54.4309214Z 
2023-01-11T21:40:54.4309547Z Generating XML reports...
2023-01-11T21:40:54.4310711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211718.xml
2023-01-11T21:40:54.4312113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4312948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4314135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4315012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4315442Z 
2023-01-11T21:40:54.4315635Z Running tests...
2023-01-11T21:40:54.4316374Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4317376Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4318314Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4318869Z 
2023-01-11T21:40:54.4319365Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4320240Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4320535Z 
2023-01-11T21:40:54.4320702Z OK (skipped=1)
2023-01-11T21:40:54.4320973Z 
2023-01-11T21:40:54.4321200Z Generating XML reports...
2023-01-11T21:40:54.4322374Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211728.xml
2023-01-11T21:40:54.4323770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4324603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4325719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4326634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4327066Z 
2023-01-11T21:40:54.4327238Z Running tests...
2023-01-11T21:40:54.4328013Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4329062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4330137Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:40:54.4330777Z 
2023-01-11T21:40:54.4331306Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4331958Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4332259Z 
2023-01-11T21:40:54.4332475Z OK (skipped=1)
2023-01-11T21:40:54.4332783Z 
2023-01-11T21:40:54.4332995Z Generating XML reports...
2023-01-11T21:40:54.4334196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211730.xml
2023-01-11T21:40:54.4335621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4336482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4337598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4338663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4339113Z 
2023-01-11T21:40:54.4339307Z Running tests...
2023-01-11T21:40:54.4340103Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4341078Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4342097Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:40:54.4342683Z 
2023-01-11T21:40:54.4343275Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4343830Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4344106Z 
2023-01-11T21:40:54.4344282Z OK (skipped=1)
2023-01-11T21:40:54.4344537Z 
2023-01-11T21:40:54.4344744Z Generating XML reports...
2023-01-11T21:40:54.4345846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211732.xml
2023-01-11T21:40:54.4347149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4347952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4349025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4349860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4350278Z 
2023-01-11T21:40:54.4350466Z Running tests...
2023-01-11T21:40:54.4351198Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4352173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4353143Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:40:54.4353724Z 
2023-01-11T21:40:54.4354208Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4354782Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4355067Z 
2023-01-11T21:40:54.4355243Z OK (skipped=1)
2023-01-11T21:40:54.4355488Z 
2023-01-11T21:40:54.4355695Z Generating XML reports...
2023-01-11T21:40:54.4356817Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211735.xml
2023-01-11T21:40:54.4358163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4358979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4360366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4361239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4361659Z 
2023-01-11T21:40:54.4361844Z Running tests...
2023-01-11T21:40:54.4362561Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4363563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4364584Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:40:54.4365183Z 
2023-01-11T21:40:54.4365655Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4366249Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4366538Z 
2023-01-11T21:40:54.4366719Z OK (skipped=1)
2023-01-11T21:40:54.4366997Z 
2023-01-11T21:40:54.4367212Z Generating XML reports...
2023-01-11T21:40:54.4368350Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211737.xml
2023-01-11T21:40:54.4369727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4370574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4371817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4372703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4373137Z 
2023-01-11T21:40:54.4373321Z Running tests...
2023-01-11T21:40:54.4374068Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4375041Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4376194Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:40:54.4376820Z 
2023-01-11T21:40:54.4377324Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4377911Z Ran 1 test in 0.003s
2023-01-11T21:40:54.4378179Z 
2023-01-11T21:40:54.4378365Z OK (skipped=1)
2023-01-11T21:40:54.4378650Z 
2023-01-11T21:40:54.4378864Z Generating XML reports...
2023-01-11T21:40:54.4380020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211740.xml
2023-01-11T21:40:54.4381391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4382252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4383381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4384287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4384703Z 
2023-01-11T21:40:54.4384893Z Running tests...
2023-01-11T21:40:54.4385640Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4386654Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4387628Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4388196Z 
2023-01-11T21:40:54.4388700Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4389294Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4389583Z 
2023-01-11T21:40:54.4389770Z OK (skipped=1)
2023-01-11T21:40:54.4390023Z 
2023-01-11T21:40:54.4390240Z Generating XML reports...
2023-01-11T21:40:54.4391420Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211742.xml
2023-01-11T21:40:54.4392821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4393656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4394786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4395697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4396132Z 
2023-01-11T21:40:54.4396325Z Running tests...
2023-01-11T21:40:54.4397067Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4398078Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4399041Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4400333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4328
2023-01-11T21:40:54.4401158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4329
2023-01-11T21:40:54.4402305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4403150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4404228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4405266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4406403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4407241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4408321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4409304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4410185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4411123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4412375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4413793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4414800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4415681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4416788Z STAGE:2023-01-11 21:17:50 4329:4329 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4417899Z STAGE:2023-01-11 21:17:50 4328:4328 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4418994Z STAGE:2023-01-11 21:17:51 4328:4328 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4420081Z STAGE:2023-01-11 21:17:51 4329:4329 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4421225Z STAGE:2023-01-11 21:17:51 4328:4328 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4422376Z STAGE:2023-01-11 21:17:51 4329:4329 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4423857Z STAGE:2023-01-11 21:17:51 4328:4328 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:17:51 4329:4329 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4424584Z 
2023-01-11T21:40:54.4425237Z STAGE:2023-01-11 21:17:51 4328:4328 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4426342Z STAGE:2023-01-11 21:17:51 4329:4329 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4427468Z STAGE:2023-01-11 21:17:51 4328:4328 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4428589Z STAGE:2023-01-11 21:17:51 4329:4329 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4429256Z ok (7.654s)
2023-01-11T21:40:54.4429517Z 
2023-01-11T21:40:54.4430014Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4430607Z Ran 1 test in 7.654s
2023-01-11T21:40:54.4430867Z 
2023-01-11T21:40:54.4431025Z OK
2023-01-11T21:40:54.4431259Z 
2023-01-11T21:40:54.4431473Z Generating XML reports...
2023-01-11T21:40:54.4432641Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211744.xml
2023-01-11T21:40:54.4433374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4433698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4434429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4434780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4434907Z 
2023-01-11T21:40:54.4435111Z Running tests...
2023-01-11T21:40:54.4435608Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4436216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4436727Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4437149Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4447
2023-01-11T21:40:54.4437639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4448
2023-01-11T21:40:54.4438359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4438686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4439677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4440059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4440791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4441114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4441854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4442205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4442681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4443119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4443905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4444687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4445137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4445565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4446215Z STAGE:2023-01-11 21:18:00 4447:4447 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4446858Z STAGE:2023-01-11 21:18:00 4448:4448 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4447508Z STAGE:2023-01-11 21:18:01 4447:4447 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4448128Z STAGE:2023-01-11 21:18:01 4448:4448 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4448801Z STAGE:2023-01-11 21:18:01 4447:4447 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4449482Z STAGE:2023-01-11 21:18:01 4448:4448 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4450506Z STAGE:2023-01-11 21:18:02 4448:4448 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:18:02 4447:4447 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4450538Z 
2023-01-11T21:40:54.4451600Z STAGE:2023-01-11 21:18:02 4447:4447 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:18:02 4448:4448 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4451637Z 
2023-01-11T21:40:54.4452753Z STAGE:2023-01-11 21:18:02 4448:4448 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:18:02 4447:4447 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4452783Z 
2023-01-11T21:40:54.4452960Z ok (7.748s)
2023-01-11T21:40:54.4452987Z 
2023-01-11T21:40:54.4453490Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4453862Z Ran 1 test in 7.748s
2023-01-11T21:40:54.4453889Z 
2023-01-11T21:40:54.4454043Z OK
2023-01-11T21:40:54.4454069Z 
2023-01-11T21:40:54.4454292Z Generating XML reports...
2023-01-11T21:40:54.4455168Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211755.xml
2023-01-11T21:40:54.4455905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4456349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4457129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4457494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4457523Z 
2023-01-11T21:40:54.4457715Z Running tests...
2023-01-11T21:40:54.4458222Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4458846Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4459382Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4459413Z 
2023-01-11T21:40:54.4459900Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4460100Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4460127Z 
2023-01-11T21:40:54.4460314Z OK (skipped=1)
2023-01-11T21:40:54.4460340Z 
2023-01-11T21:40:54.4460568Z Generating XML reports...
2023-01-11T21:40:54.4461460Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211805.xml
2023-01-11T21:40:54.4462201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4462533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4463301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4463666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4463697Z 
2023-01-11T21:40:54.4463866Z Running tests...
2023-01-11T21:40:54.4464370Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4464994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4465506Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4465535Z 
2023-01-11T21:40:54.4466040Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4466238Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4466266Z 
2023-01-11T21:40:54.4466452Z OK (skipped=1)
2023-01-11T21:40:54.4466479Z 
2023-01-11T21:40:54.4466700Z Generating XML reports...
2023-01-11T21:40:54.4467582Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211807.xml
2023-01-11T21:40:54.4468326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4468654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4469416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4469781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4469810Z 
2023-01-11T21:40:54.4470003Z Running tests...
2023-01-11T21:40:54.4470511Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4471132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4471660Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4472166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4632
2023-01-11T21:40:54.4472589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4633
2023-01-11T21:40:54.4473341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4473670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4474516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4474900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4475648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4475977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4476749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4477088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4477557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4478028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4478832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4479935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4480400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4480850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4481521Z STAGE:2023-01-11 21:18:15 4633:4633 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4482148Z STAGE:2023-01-11 21:18:15 4632:4632 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4482805Z STAGE:2023-01-11 21:18:16 4632:4632 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4483446Z STAGE:2023-01-11 21:18:16 4633:4633 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4484132Z STAGE:2023-01-11 21:18:16 4632:4632 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4484819Z STAGE:2023-01-11 21:18:16 4633:4633 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4485000Z ok (7.718s)
2023-01-11T21:40:54.4485029Z 
2023-01-11T21:40:54.4485533Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4485733Z Ran 1 test in 7.719s
2023-01-11T21:40:54.4485774Z 
2023-01-11T21:40:54.4485934Z OK
2023-01-11T21:40:54.4485961Z 
2023-01-11T21:40:54.4486164Z Generating XML reports...
2023-01-11T21:40:54.4487063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211810.xml
2023-01-11T21:40:54.4487811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4488142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4488909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4489292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4489324Z 
2023-01-11T21:40:54.4489510Z Running tests...
2023-01-11T21:40:54.4490020Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4490590Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4491248Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4491649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4751
2023-01-11T21:40:54.4492038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4752
2023-01-11T21:40:54.4492748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4493168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4493917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4494260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4494963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4495267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4496070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4496420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4496871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4497327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4498102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4498875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4499305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4499735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4500352Z STAGE:2023-01-11 21:18:25 4751:4751 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4500980Z STAGE:2023-01-11 21:18:25 4752:4752 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4501618Z STAGE:2023-01-11 21:18:27 4751:4751 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4502256Z STAGE:2023-01-11 21:18:27 4752:4752 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4502921Z STAGE:2023-01-11 21:18:27 4751:4751 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4503586Z STAGE:2023-01-11 21:18:27 4752:4752 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4503761Z ok (7.747s)
2023-01-11T21:40:54.4503789Z 
2023-01-11T21:40:54.4504307Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4504481Z Ran 1 test in 7.748s
2023-01-11T21:40:54.4504535Z 
2023-01-11T21:40:54.4504666Z OK
2023-01-11T21:40:54.4504692Z 
2023-01-11T21:40:54.4504915Z Generating XML reports...
2023-01-11T21:40:54.4505793Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211820.xml
2023-01-11T21:40:54.4506528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4506863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4507615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4507968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4507998Z 
2023-01-11T21:40:54.4508188Z Running tests...
2023-01-11T21:40:54.4508774Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4509404Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4509903Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4511499Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2023-01-11T21:40:54.4511535Z 
2023-01-11T21:40:54.4512059Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4512257Z Ran 1 test in 1.637s
2023-01-11T21:40:54.4512286Z 
2023-01-11T21:40:54.4512471Z OK (skipped=1)
2023-01-11T21:40:54.4512498Z 
2023-01-11T21:40:54.4512714Z Generating XML reports...
2023-01-11T21:40:54.4513685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211830.xml
2023-01-11T21:40:54.4514426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4514736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4515495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4515864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4515895Z 
2023-01-11T21:40:54.4516090Z Running tests...
2023-01-11T21:40:54.4516604Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4517231Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4517754Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4519263Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2023-01-11T21:40:54.4519294Z 
2023-01-11T21:40:54.4520110Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4520292Z Ran 1 test in 1.640s
2023-01-11T21:40:54.4520342Z 
2023-01-11T21:40:54.4520517Z OK (skipped=1)
2023-01-11T21:40:54.4520544Z 
2023-01-11T21:40:54.4520768Z Generating XML reports...
2023-01-11T21:40:54.4521662Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211834.xml
2023-01-11T21:40:54.4522407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4522747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4523508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4523871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4523901Z 
2023-01-11T21:40:54.4524090Z Running tests...
2023-01-11T21:40:54.4524570Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4525197Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4525724Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4526139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4938
2023-01-11T21:40:54.4526543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4939
2023-01-11T21:40:54.4527269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4527748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4528521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4528882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4529620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4530053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4530828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4531191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4531657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4532430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4532884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4533676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4534108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4534571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4534761Z ok (6.946s)
2023-01-11T21:40:54.4534790Z 
2023-01-11T21:40:54.4535296Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4535513Z Ran 1 test in 6.946s
2023-01-11T21:40:54.4535545Z 
2023-01-11T21:40:54.4535714Z OK
2023-01-11T21:40:54.4535752Z 
2023-01-11T21:40:54.4535985Z Generating XML reports...
2023-01-11T21:40:54.4536899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211839.xml
2023-01-11T21:40:54.4537646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4537965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4538741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4539127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4539155Z 
2023-01-11T21:40:54.4539365Z Running tests...
2023-01-11T21:40:54.4539888Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4540502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4541038Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4541469Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5053
2023-01-11T21:40:54.4541870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5054
2023-01-11T21:40:54.4542604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4542949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4543714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4544070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4544758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4545071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4545956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4546326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4546784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4547251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4548163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4548994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4549436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4549874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4550353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.4550830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.4551638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4552405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4552885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.4553353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.4554157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.4554944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.4555415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:40:54.4555873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:40:54.4556756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T21:40:54.4557573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T21:40:54.4557739Z ok (8.435s)
2023-01-11T21:40:54.4557770Z 
2023-01-11T21:40:54.4558289Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4558512Z Ran 1 test in 8.435s
2023-01-11T21:40:54.4558538Z 
2023-01-11T21:40:54.4558710Z OK
2023-01-11T21:40:54.4558745Z 
2023-01-11T21:40:54.4558989Z Generating XML reports...
2023-01-11T21:40:54.4560624Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211848.xml
2023-01-11T21:40:54.4561141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4561326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4561727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4561904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4561927Z 
2023-01-11T21:40:54.4562038Z Running tests...
2023-01-11T21:40:54.4562305Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4562620Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4563055Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4563276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5204
2023-01-11T21:40:54.4563492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5205
2023-01-11T21:40:54.4563870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4564026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4564481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4564681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4565047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4565217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4565596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4565785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4566032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4566262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4566669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4567070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4567300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4567526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4567633Z ok (6.920s)
2023-01-11T21:40:54.4567653Z 
2023-01-11T21:40:54.4567920Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4568034Z Ran 1 test in 6.920s
2023-01-11T21:40:54.4568054Z 
2023-01-11T21:40:54.4568145Z OK
2023-01-11T21:40:54.4568164Z 
2023-01-11T21:40:54.4568271Z Generating XML reports...
2023-01-11T21:40:54.4568729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211859.xml
2023-01-11T21:40:54.4569102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4569278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4569657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4569847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4569870Z 
2023-01-11T21:40:54.4569979Z Running tests...
2023-01-11T21:40:54.4570241Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4570555Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4570960Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4570980Z 
2023-01-11T21:40:54.4571245Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4571356Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4571375Z 
2023-01-11T21:40:54.4571481Z OK (skipped=1)
2023-01-11T21:40:54.4571501Z 
2023-01-11T21:40:54.4571623Z Generating XML reports...
2023-01-11T21:40:54.4572073Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211908.xml
2023-01-11T21:40:54.4572509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4572686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4573065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4573239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4573259Z 
2023-01-11T21:40:54.4573368Z Running tests...
2023-01-11T21:40:54.4573673Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4573997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4574414Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4574434Z 
2023-01-11T21:40:54.4574693Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4574809Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4574829Z 
2023-01-11T21:40:54.4574935Z OK (skipped=1)
2023-01-11T21:40:54.4574954Z 
2023-01-11T21:40:54.4575077Z Generating XML reports...
2023-01-11T21:40:54.4575507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211911.xml
2023-01-11T21:40:54.4575875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4576054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4576431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4576621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4576640Z 
2023-01-11T21:40:54.4576747Z Running tests...
2023-01-11T21:40:54.4577010Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4577321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4577732Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4577770Z 
2023-01-11T21:40:54.4578010Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4578121Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4578141Z 
2023-01-11T21:40:54.4578250Z OK (skipped=1)
2023-01-11T21:40:54.4578268Z 
2023-01-11T21:40:54.4578391Z Generating XML reports...
2023-01-11T21:40:54.4578845Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211913.xml
2023-01-11T21:40:54.4579213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4579392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4579771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4579944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4579982Z 
2023-01-11T21:40:54.4580074Z Running tests...
2023-01-11T21:40:54.4580334Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4580647Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4581067Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4581087Z 
2023-01-11T21:40:54.4581344Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4581453Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4581472Z 
2023-01-11T21:40:54.4581639Z OK (skipped=1)
2023-01-11T21:40:54.4581659Z 
2023-01-11T21:40:54.4581782Z Generating XML reports...
2023-01-11T21:40:54.4582217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211916.xml
2023-01-11T21:40:54.4582587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4582758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4583176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4583370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4583389Z 
2023-01-11T21:40:54.4583497Z Running tests...
2023-01-11T21:40:54.4583758Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4584074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4584489Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4584509Z 
2023-01-11T21:40:54.4584751Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4584863Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4584882Z 
2023-01-11T21:40:54.4584988Z OK (skipped=1)
2023-01-11T21:40:54.4585007Z 
2023-01-11T21:40:54.4585130Z Generating XML reports...
2023-01-11T21:40:54.4585581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211918.xml
2023-01-11T21:40:54.4585952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4586126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4586506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4586683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4586720Z 
2023-01-11T21:40:54.4586810Z Running tests...
2023-01-11T21:40:54.4587070Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4587381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4587794Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4587814Z 
2023-01-11T21:40:54.4588071Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4588181Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4588200Z 
2023-01-11T21:40:54.4588305Z OK (skipped=1)
2023-01-11T21:40:54.4588323Z 
2023-01-11T21:40:54.4588445Z Generating XML reports...
2023-01-11T21:40:54.4588877Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211920.xml
2023-01-11T21:40:54.4589253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4589427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4589806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4589995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4590014Z 
2023-01-11T21:40:54.4590123Z Running tests...
2023-01-11T21:40:54.4590382Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4590695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4591111Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4591183Z 
2023-01-11T21:40:54.4591431Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4591543Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4591562Z 
2023-01-11T21:40:54.4591670Z OK (skipped=1)
2023-01-11T21:40:54.4591689Z 
2023-01-11T21:40:54.4591810Z Generating XML reports...
2023-01-11T21:40:54.4592254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211923.xml
2023-01-11T21:40:54.4592665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4592846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4593228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4593417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4593441Z 
2023-01-11T21:40:54.4593532Z Running tests...
2023-01-11T21:40:54.4593794Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4594109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4594519Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4594538Z 
2023-01-11T21:40:54.4594795Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4594908Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4594928Z 
2023-01-11T21:40:54.4595033Z OK (skipped=1)
2023-01-11T21:40:54.4595052Z 
2023-01-11T21:40:54.4595174Z Generating XML reports...
2023-01-11T21:40:54.4595621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211925.xml
2023-01-11T21:40:54.4595972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4596151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4596533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4596721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4596741Z 
2023-01-11T21:40:54.4596847Z Running tests...
2023-01-11T21:40:54.4597107Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4597421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4597820Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4597840Z 
2023-01-11T21:40:54.4598096Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4598191Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4598214Z 
2023-01-11T21:40:54.4598321Z OK (skipped=1)
2023-01-11T21:40:54.4598340Z 
2023-01-11T21:40:54.4598462Z Generating XML reports...
2023-01-11T21:40:54.4598905Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211928.xml
2023-01-11T21:40:54.4599269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4599716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4600119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4600309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4600329Z 
2023-01-11T21:40:54.4600420Z Running tests...
2023-01-11T21:40:54.4600677Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4600986Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4601379Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4601401Z 
2023-01-11T21:40:54.4601660Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4601771Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4601791Z 
2023-01-11T21:40:54.4601897Z OK (skipped=1)
2023-01-11T21:40:54.4601916Z 
2023-01-11T21:40:54.4602037Z Generating XML reports...
2023-01-11T21:40:54.4602541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211930.xml
2023-01-11T21:40:54.4602908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4603083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4603460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4603653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4603673Z 
2023-01-11T21:40:54.4603780Z Running tests...
2023-01-11T21:40:54.4604039Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4604350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4604749Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4604768Z 
2023-01-11T21:40:54.4605025Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4605119Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4605139Z 
2023-01-11T21:40:54.4605245Z OK (skipped=1)
2023-01-11T21:40:54.4605264Z 
2023-01-11T21:40:54.4605385Z Generating XML reports...
2023-01-11T21:40:54.4605833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211932.xml
2023-01-11T21:40:54.4606200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4606375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4606750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4606940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4606960Z 
2023-01-11T21:40:54.4607067Z Running tests...
2023-01-11T21:40:54.4607310Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4607619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4608024Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4608047Z 
2023-01-11T21:40:54.4608302Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4608411Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4608430Z 
2023-01-11T21:40:54.4608534Z OK (skipped=1)
2023-01-11T21:40:54.4608553Z 
2023-01-11T21:40:54.4608675Z Generating XML reports...
2023-01-11T21:40:54.4609122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211935.xml
2023-01-11T21:40:54.4609477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4609654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4610033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4610221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4610292Z 
2023-01-11T21:40:54.4610406Z Running tests...
2023-01-11T21:40:54.4610670Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4610982Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4611375Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.4611395Z 
2023-01-11T21:40:54.4611708Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4611807Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4611827Z 
2023-01-11T21:40:54.4611935Z OK (skipped=1)
2023-01-11T21:40:54.4611954Z 
2023-01-11T21:40:54.4612077Z Generating XML reports...
2023-01-11T21:40:54.4612526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211937.xml
2023-01-11T21:40:54.4612892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4613116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4613496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4613684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4613704Z 
2023-01-11T21:40:54.4613810Z Running tests...
2023-01-11T21:40:54.4614058Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4614368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4614657Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4614676Z 
2023-01-11T21:40:54.4614931Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4615045Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4615064Z 
2023-01-11T21:40:54.4615169Z OK (skipped=1)
2023-01-11T21:40:54.4615188Z 
2023-01-11T21:40:54.4615309Z Generating XML reports...
2023-01-11T21:40:54.4615753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211940.xml
2023-01-11T21:40:54.4616122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4616280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4616659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4616848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4616867Z 
2023-01-11T21:40:54.4616973Z Running tests...
2023-01-11T21:40:54.4617230Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4617544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4617816Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4617836Z 
2023-01-11T21:40:54.4618090Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4618184Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4618220Z 
2023-01-11T21:40:54.4618311Z OK (skipped=1)
2023-01-11T21:40:54.4618330Z 
2023-01-11T21:40:54.4618456Z Generating XML reports...
2023-01-11T21:40:54.4618904Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211942.xml
2023-01-11T21:40:54.4619274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4619448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4619887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4620078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4620097Z 
2023-01-11T21:40:54.4620204Z Running tests...
2023-01-11T21:40:54.4620446Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4620757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4621069Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4621091Z 
2023-01-11T21:40:54.4621356Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4621467Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4621486Z 
2023-01-11T21:40:54.4621592Z OK (skipped=1)
2023-01-11T21:40:54.4621611Z 
2023-01-11T21:40:54.4621732Z Generating XML reports...
2023-01-11T21:40:54.4622183Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211944.xml
2023-01-11T21:40:54.4622553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4622713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4623091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4623283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4623303Z 
2023-01-11T21:40:54.4623409Z Running tests...
2023-01-11T21:40:54.4623669Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4623980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4624258Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4624280Z 
2023-01-11T21:40:54.4624536Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4624646Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4624666Z 
2023-01-11T21:40:54.4624755Z OK (skipped=1)
2023-01-11T21:40:54.4624775Z 
2023-01-11T21:40:54.4624897Z Generating XML reports...
2023-01-11T21:40:54.4625341Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211947.xml
2023-01-11T21:40:54.4625713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4625887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4626263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4626452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4626474Z 
2023-01-11T21:40:54.4626580Z Running tests...
2023-01-11T21:40:54.4626822Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4627135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4627404Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4627424Z 
2023-01-11T21:40:54.4627680Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4627790Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4627809Z 
2023-01-11T21:40:54.4627916Z OK (skipped=1)
2023-01-11T21:40:54.4627935Z 
2023-01-11T21:40:54.4628055Z Generating XML reports...
2023-01-11T21:40:54.4628501Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211949.xml
2023-01-11T21:40:54.4628870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4629083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4629466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4629653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4629673Z 
2023-01-11T21:40:54.4629780Z Running tests...
2023-01-11T21:40:54.4630080Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4630400Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4630668Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4630688Z 
2023-01-11T21:40:54.4630946Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4631062Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4631082Z 
2023-01-11T21:40:54.4631171Z OK (skipped=1)
2023-01-11T21:40:54.4631190Z 
2023-01-11T21:40:54.4631313Z Generating XML reports...
2023-01-11T21:40:54.4631757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211952.xml
2023-01-11T21:40:54.4632123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4632299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4632679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4632869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4632889Z 
2023-01-11T21:40:54.4632995Z Running tests...
2023-01-11T21:40:54.4633238Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4633552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4633818Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4633837Z 
2023-01-11T21:40:54.4634092Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4634201Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4634220Z 
2023-01-11T21:40:54.4634326Z OK (skipped=1)
2023-01-11T21:40:54.4634345Z 
2023-01-11T21:40:54.4634467Z Generating XML reports...
2023-01-11T21:40:54.4634911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211954.xml
2023-01-11T21:40:54.4635278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4635434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4635815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4636003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4636022Z 
2023-01-11T21:40:54.4636127Z Running tests...
2023-01-11T21:40:54.4636387Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4636696Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4636972Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4636991Z 
2023-01-11T21:40:54.4637250Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4637359Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4637378Z 
2023-01-11T21:40:54.4637467Z OK (skipped=1)
2023-01-11T21:40:54.4637486Z 
2023-01-11T21:40:54.4637609Z Generating XML reports...
2023-01-11T21:40:54.4638114Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211956.xml
2023-01-11T21:40:54.4638483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4638656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4639032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4639264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4639286Z 
2023-01-11T21:40:54.4639634Z Running tests...
2023-01-11T21:40:54.4639912Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4640210Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4640480Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4640506Z 
2023-01-11T21:40:54.4640763Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4640873Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4640892Z 
2023-01-11T21:40:54.4640998Z OK (skipped=1)
2023-01-11T21:40:54.4641017Z 
2023-01-11T21:40:54.4641138Z Generating XML reports...
2023-01-11T21:40:54.4641581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211959.xml
2023-01-11T21:40:54.4641952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4642110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4642486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4642675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4642697Z 
2023-01-11T21:40:54.4642804Z Running tests...
2023-01-11T21:40:54.4643062Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4643375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4643634Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4643654Z 
2023-01-11T21:40:54.4643908Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4644023Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4644043Z 
2023-01-11T21:40:54.4644133Z OK (skipped=1)
2023-01-11T21:40:54.4644152Z 
2023-01-11T21:40:54.4644277Z Generating XML reports...
2023-01-11T21:40:54.4644722Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212001.xml
2023-01-11T21:40:54.4645089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4645266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4645641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4645834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4645854Z 
2023-01-11T21:40:54.4645962Z Running tests...
2023-01-11T21:40:54.4646222Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4646520Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4646782Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4646801Z 
2023-01-11T21:40:54.4647055Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4647164Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4647282Z 
2023-01-11T21:40:54.4647398Z OK (skipped=1)
2023-01-11T21:40:54.4647417Z 
2023-01-11T21:40:54.4647541Z Generating XML reports...
2023-01-11T21:40:54.4647989Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212004.xml
2023-01-11T21:40:54.4648357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4648531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4648957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4649156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4649175Z 
2023-01-11T21:40:54.4649282Z Running tests...
2023-01-11T21:40:54.4649542Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4649854Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4650137Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T21:40:54.4650156Z 
2023-01-11T21:40:54.4650417Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4650528Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4650547Z 
2023-01-11T21:40:54.4650636Z OK (skipped=1)
2023-01-11T21:40:54.4650672Z 
2023-01-11T21:40:54.4650778Z Generating XML reports...
2023-01-11T21:40:54.4651227Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212006.xml
2023-01-11T21:40:54.4651600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4651773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4652150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4652344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4652363Z 
2023-01-11T21:40:54.4652469Z Running tests...
2023-01-11T21:40:54.4652729Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4653024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4653314Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T21:40:54.4653335Z 
2023-01-11T21:40:54.4653592Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4653701Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4653720Z 
2023-01-11T21:40:54.4653824Z OK (skipped=1)
2023-01-11T21:40:54.4653843Z 
2023-01-11T21:40:54.4653964Z Generating XML reports...
2023-01-11T21:40:54.4654403Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212008.xml
2023-01-11T21:40:54.4654774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4654949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4655308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4655497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4655520Z 
2023-01-11T21:40:54.4655626Z Running tests...
2023-01-11T21:40:54.4655883Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4656193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4656458Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4656527Z 
2023-01-11T21:40:54.4656792Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4656904Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4656923Z 
2023-01-11T21:40:54.4657013Z OK (skipped=1)
2023-01-11T21:40:54.4657050Z 
2023-01-11T21:40:54.4657156Z Generating XML reports...
2023-01-11T21:40:54.4657597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212011.xml
2023-01-11T21:40:54.4658003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4658184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4658560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4658749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4658772Z 
2023-01-11T21:40:54.4658879Z Running tests...
2023-01-11T21:40:54.4659136Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4659426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4659689Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4659906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6210
2023-01-11T21:40:54.4660123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6211
2023-01-11T21:40:54.4660496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4660668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4661041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4661236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4661599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4661754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4662123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4662309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4662559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4662805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4663206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4663599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4663832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4664044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4664146Z ok (6.944s)
2023-01-11T21:40:54.4664166Z 
2023-01-11T21:40:54.4664428Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4664538Z Ran 1 test in 6.944s
2023-01-11T21:40:54.4664557Z 
2023-01-11T21:40:54.4664651Z OK
2023-01-11T21:40:54.4664671Z 
2023-01-11T21:40:54.4664794Z Generating XML reports...
2023-01-11T21:40:54.4665245Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212013.xml
2023-01-11T21:40:54.4665612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4665841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4666207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4666397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4666417Z 
2023-01-11T21:40:54.4666524Z Running tests...
2023-01-11T21:40:54.4666782Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4667132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4667401Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4667421Z 
2023-01-11T21:40:54.4667680Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4667791Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4667810Z 
2023-01-11T21:40:54.4667915Z OK (skipped=1)
2023-01-11T21:40:54.4667938Z 
2023-01-11T21:40:54.4668044Z Generating XML reports...
2023-01-11T21:40:54.4668489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212023.xml
2023-01-11T21:40:54.4668853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4669027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4669407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4669598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4669617Z 
2023-01-11T21:40:54.4669725Z Running tests...
2023-01-11T21:40:54.4669985Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4670280Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4670551Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4670571Z 
2023-01-11T21:40:54.4670830Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4670939Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4670959Z 
2023-01-11T21:40:54.4671065Z OK (skipped=1)
2023-01-11T21:40:54.4671084Z 
2023-01-11T21:40:54.4671205Z Generating XML reports...
2023-01-11T21:40:54.4671645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212025.xml
2023-01-11T21:40:54.4672010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4672183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4672542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4672736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4672756Z 
2023-01-11T21:40:54.4672862Z Running tests...
2023-01-11T21:40:54.4673120Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4673430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4673703Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4673723Z 
2023-01-11T21:40:54.4673983Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4674096Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4674115Z 
2023-01-11T21:40:54.4674222Z OK (skipped=1)
2023-01-11T21:40:54.4674241Z 
2023-01-11T21:40:54.4674346Z Generating XML reports...
2023-01-11T21:40:54.4674791Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212027.xml
2023-01-11T21:40:54.4675215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4675388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4675765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4675955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4675975Z 
2023-01-11T21:40:54.4676083Z Running tests...
2023-01-11T21:40:54.4676396Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4676702Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4676964Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4677181Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6424
2023-01-11T21:40:54.4677399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6425
2023-01-11T21:40:54.4677768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4677941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4678315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4678507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4678870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4679028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4679617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4679820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4680069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4680313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4680720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4681121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4681351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4681579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4681895Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4682227Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4682559Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4682882Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4683222Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4683565Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4683888Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4684208Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4684517Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4684944Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4685268Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4685603Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4685921Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4686294Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4686633Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4686969Z STAGE:2023-01-11 21:20:35 6424:6424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4687292Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4687615Z STAGE:2023-01-11 21:20:35 6425:6425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4687720Z ok (6.262s)
2023-01-11T21:40:54.4687740Z 
2023-01-11T21:40:54.4688000Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4688109Z Ran 1 test in 6.262s
2023-01-11T21:40:54.4688129Z 
2023-01-11T21:40:54.4688219Z OK
2023-01-11T21:40:54.4688238Z 
2023-01-11T21:40:54.4688360Z Generating XML reports...
2023-01-11T21:40:54.4688814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212030.xml
2023-01-11T21:40:54.4689184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4689357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4689720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4689916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4689935Z 
2023-01-11T21:40:54.4690043Z Running tests...
2023-01-11T21:40:54.4690303Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4690617Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4690884Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4691105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6542
2023-01-11T21:40:54.4691319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6543
2023-01-11T21:40:54.4691673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4691849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4692229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4692419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4692779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4692949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4693328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4693515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4693760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4693987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4694452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4694847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4695075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4695300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4695704Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4696040Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4696371Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4696700Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4697028Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4697369Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4697687Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4698005Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4698334Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4698674Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4699000Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4699336Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4699640Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4699958Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4700283Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4700620Z STAGE:2023-01-11 21:20:44 6543:6543 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4700946Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4701282Z STAGE:2023-01-11 21:20:44 6542:6542 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4701384Z ok (6.316s)
2023-01-11T21:40:54.4701404Z 
2023-01-11T21:40:54.4701666Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4701781Z Ran 1 test in 6.316s
2023-01-11T21:40:54.4701801Z 
2023-01-11T21:40:54.4701876Z OK
2023-01-11T21:40:54.4701894Z 
2023-01-11T21:40:54.4702019Z Generating XML reports...
2023-01-11T21:40:54.4702471Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212039.xml
2023-01-11T21:40:54.4702842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4703016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4703398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4703588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4703608Z 
2023-01-11T21:40:54.4703716Z Running tests...
2023-01-11T21:40:54.4703957Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4704273Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4704606Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4704823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6660
2023-01-11T21:40:54.4705038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6661
2023-01-11T21:40:54.4705412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4705629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4706014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4706204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4706549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4706726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4707097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4707284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4707529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4707775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4708174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4708570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4708801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4709016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4709347Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4709671Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4710000Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4710332Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4710671Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4711012Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4711330Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4711656Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4711965Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4712302Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4712624Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4713004Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4713332Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4713652Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4713977Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4714362Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4714684Z STAGE:2023-01-11 21:20:53 6660:6660 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4715023Z STAGE:2023-01-11 21:20:53 6661:6661 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4715126Z ok (6.345s)
2023-01-11T21:40:54.4715146Z 
2023-01-11T21:40:54.4715463Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4715583Z Ran 1 test in 6.345s
2023-01-11T21:40:54.4715603Z 
2023-01-11T21:40:54.4715695Z OK
2023-01-11T21:40:54.4715714Z 
2023-01-11T21:40:54.4715837Z Generating XML reports...
2023-01-11T21:40:54.4716291Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212047.xml
2023-01-11T21:40:54.4716663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4716827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4717209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4717399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4717418Z 
2023-01-11T21:40:54.4717525Z Running tests...
2023-01-11T21:40:54.4717788Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4718103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4718349Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:40:54.4718368Z 
2023-01-11T21:40:54.4718625Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4718723Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4718758Z 
2023-01-11T21:40:54.4718849Z OK (skipped=1)
2023-01-11T21:40:54.4718868Z 
2023-01-11T21:40:54.4718991Z Generating XML reports...
2023-01-11T21:40:54.4719656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212056.xml
2023-01-11T21:40:54.4720045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4720220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4720602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4720792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4720812Z 
2023-01-11T21:40:54.4720920Z Running tests...
2023-01-11T21:40:54.4721165Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4721484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4721739Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:40:54.4721758Z 
2023-01-11T21:40:54.4722017Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4722127Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4722147Z 
2023-01-11T21:40:54.4722252Z OK (skipped=1)
2023-01-11T21:40:54.4722271Z 
2023-01-11T21:40:54.4722393Z Generating XML reports...
2023-01-11T21:40:54.4722843Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212059.xml
2023-01-11T21:40:54.4723214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4723372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4723749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4724023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4724044Z 
2023-01-11T21:40:54.4724151Z Running tests...
2023-01-11T21:40:54.4724415Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4724726Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4725034Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4725257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6844
2023-01-11T21:40:54.4725458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6845
2023-01-11T21:40:54.4725830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4726007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4726382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4726567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4726924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4727094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4727466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4727655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4727882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4728126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4728527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4728918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4729145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4729369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4729473Z ok (6.854s)
2023-01-11T21:40:54.4729493Z 
2023-01-11T21:40:54.4729750Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4729854Z Ran 1 test in 6.854s
2023-01-11T21:40:54.4729874Z 
2023-01-11T21:40:54.4729949Z OK
2023-01-11T21:40:54.4729968Z 
2023-01-11T21:40:54.4730085Z Generating XML reports...
2023-01-11T21:40:54.4730531Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212101.xml
2023-01-11T21:40:54.4730901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4731074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4731445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4731629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4731649Z 
2023-01-11T21:40:54.4731757Z Running tests...
2023-01-11T21:40:54.4732001Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4732308Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4732571Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4732840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6961
2023-01-11T21:40:54.4733048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6962
2023-01-11T21:40:54.4733416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4733586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4733956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4734183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4734536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4734704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4735068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4735251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4735492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4735730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4736125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4736520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4736732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4736956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4737053Z ok (7.133s)
2023-01-11T21:40:54.4737076Z 
2023-01-11T21:40:54.4737333Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4737442Z Ran 1 test in 7.134s
2023-01-11T21:40:54.4737461Z 
2023-01-11T21:40:54.4737553Z OK
2023-01-11T21:40:54.4737572Z 
2023-01-11T21:40:54.4737690Z Generating XML reports...
2023-01-11T21:40:54.4738135Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212110.xml
2023-01-11T21:40:54.4738500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4738662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4739041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4739229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4739248Z 
2023-01-11T21:40:54.4739353Z Running tests...
2023-01-11T21:40:54.4739616Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4739927Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4740180Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:40:54.4740200Z 
2023-01-11T21:40:54.4740449Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4740555Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4740574Z 
2023-01-11T21:40:54.4740666Z OK (skipped=1)
2023-01-11T21:40:54.4740688Z 
2023-01-11T21:40:54.4740810Z Generating XML reports...
2023-01-11T21:40:54.4741254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212120.xml
2023-01-11T21:40:54.4741622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4741849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4742230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4742416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4742436Z 
2023-01-11T21:40:54.4742536Z Running tests...
2023-01-11T21:40:54.4742781Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4743145Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4743416Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4743632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7111
2023-01-11T21:40:54.4743842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7112
2023-01-11T21:40:54.4744210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4744388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4744763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4744949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4745292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4745468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4745837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4746022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4746266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4746513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4746912Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4747306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4747529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4747755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.4747976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4748207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.4748599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4748990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4749083Z ok (7.021s)
2023-01-11T21:40:54.4749103Z 
2023-01-11T21:40:54.4749360Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4749464Z Ran 1 test in 7.021s
2023-01-11T21:40:54.4749484Z 
2023-01-11T21:40:54.4749558Z OK
2023-01-11T21:40:54.4749589Z 
2023-01-11T21:40:54.4749696Z Generating XML reports...
2023-01-11T21:40:54.4750142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212122.xml
2023-01-11T21:40:54.4750509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4750675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4751107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4751291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4751311Z 
2023-01-11T21:40:54.4751414Z Running tests...
2023-01-11T21:40:54.4751666Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4751960Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4752248Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:40:54.4752269Z 
2023-01-11T21:40:54.4752527Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4752634Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4752653Z 
2023-01-11T21:40:54.4752754Z OK (skipped=1)
2023-01-11T21:40:54.4752773Z 
2023-01-11T21:40:54.4752888Z Generating XML reports...
2023-01-11T21:40:54.4753332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212132.xml
2023-01-11T21:40:54.4753695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4753863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4754226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4754415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4754435Z 
2023-01-11T21:40:54.4754533Z Running tests...
2023-01-11T21:40:54.4754787Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4755092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4755346Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4755561Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7263
2023-01-11T21:40:54.4755770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7264
2023-01-11T21:40:54.4756122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4756292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4756663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4756846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4757202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4757367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4757733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4757920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4758164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4758391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4758783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4759171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4759612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4759847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4760077Z skip: Skipped due to small world size. (4.224s)
2023-01-11T21:40:54.4760097Z 
2023-01-11T21:40:54.4760364Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4760469Z Ran 1 test in 4.224s
2023-01-11T21:40:54.4760489Z 
2023-01-11T21:40:54.4760578Z OK (skipped=1)
2023-01-11T21:40:54.4760609Z 
2023-01-11T21:40:54.4760717Z Generating XML reports...
2023-01-11T21:40:54.4761162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212134.xml
2023-01-11T21:40:54.4761582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4761764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4762139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4762321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4762344Z 
2023-01-11T21:40:54.4762446Z Running tests...
2023-01-11T21:40:54.4762698Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4762995Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4763276Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4763296Z 
2023-01-11T21:40:54.4763548Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4763658Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4763678Z 
2023-01-11T21:40:54.4763781Z OK (skipped=1)
2023-01-11T21:40:54.4763800Z 
2023-01-11T21:40:54.4763921Z Generating XML reports...
2023-01-11T21:40:54.4764364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212141.xml
2023-01-11T21:40:54.4764727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4764903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4765265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4765455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4765475Z 
2023-01-11T21:40:54.4765575Z Running tests...
2023-01-11T21:40:54.4765833Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4766138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4766427Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4766447Z 
2023-01-11T21:40:54.4766701Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4766813Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4766832Z 
2023-01-11T21:40:54.4766935Z OK (skipped=1)
2023-01-11T21:40:54.4766954Z 
2023-01-11T21:40:54.4767058Z Generating XML reports...
2023-01-11T21:40:54.4767502Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212144.xml
2023-01-11T21:40:54.4767869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4768040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4768417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4768605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4768624Z 
2023-01-11T21:40:54.4768732Z Running tests...
2023-01-11T21:40:54.4768987Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4769342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4769617Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4770415Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2023-01-11T21:40:54.4770438Z 
2023-01-11T21:40:54.4770701Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4770809Z Ran 1 test in 1.643s
2023-01-11T21:40:54.4770828Z 
2023-01-11T21:40:54.4770935Z OK (skipped=1)
2023-01-11T21:40:54.4770954Z 
2023-01-11T21:40:54.4771074Z Generating XML reports...
2023-01-11T21:40:54.4771519Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212146.xml
2023-01-11T21:40:54.4771891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4772060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4772422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4772609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4772629Z 
2023-01-11T21:40:54.4772738Z Running tests...
2023-01-11T21:40:54.4772992Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4773300Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4773582Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4773801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7468
2023-01-11T21:40:54.4774015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7469
2023-01-11T21:40:54.4774381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4774538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4774903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4775074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4775447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4775634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4776008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4776199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4776443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4776670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4777066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4777460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4777689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4777909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4778236Z STAGE:2023-01-11 21:21:56 7469:7469 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4778624Z STAGE:2023-01-11 21:21:56 7468:7468 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.4778945Z STAGE:2023-01-11 21:21:57 7468:7468 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4779270Z STAGE:2023-01-11 21:21:57 7469:7469 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.4779594Z STAGE:2023-01-11 21:21:57 7469:7469 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4779976Z STAGE:2023-01-11 21:21:57 7468:7468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.4780082Z ok (7.756s)
2023-01-11T21:40:54.4780101Z 
2023-01-11T21:40:54.4780363Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4780472Z Ran 1 test in 7.756s
2023-01-11T21:40:54.4780492Z 
2023-01-11T21:40:54.4780577Z OK
2023-01-11T21:40:54.4780600Z 
2023-01-11T21:40:54.4780719Z Generating XML reports...
2023-01-11T21:40:54.4781167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212150.xml
2023-01-11T21:40:54.4781535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4781695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4782078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4782270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4782289Z 
2023-01-11T21:40:54.4782396Z Running tests...
2023-01-11T21:40:54.4782654Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4782965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4783264Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4783284Z 
2023-01-11T21:40:54.4783543Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4783636Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4783672Z 
2023-01-11T21:40:54.4783762Z OK (skipped=1)
2023-01-11T21:40:54.4783781Z 
2023-01-11T21:40:54.4783900Z Generating XML reports...
2023-01-11T21:40:54.4784352Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212200.xml
2023-01-11T21:40:54.4784719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4784893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4785270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4785462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4785483Z 
2023-01-11T21:40:54.4785589Z Running tests...
2023-01-11T21:40:54.4785833Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4786143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4786435Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4787182Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s)
2023-01-11T21:40:54.4787203Z 
2023-01-11T21:40:54.4787459Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4787627Z Ran 1 test in 1.617s
2023-01-11T21:40:54.4787647Z 
2023-01-11T21:40:54.4787752Z OK (skipped=1)
2023-01-11T21:40:54.4787771Z 
2023-01-11T21:40:54.4787894Z Generating XML reports...
2023-01-11T21:40:54.4788341Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212203.xml
2023-01-11T21:40:54.4788710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4788912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4789302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4789493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4789513Z 
2023-01-11T21:40:54.4789620Z Running tests...
2023-01-11T21:40:54.4789878Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4790195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4790485Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4790505Z 
2023-01-11T21:40:54.4790762Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4790872Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4790891Z 
2023-01-11T21:40:54.4790980Z OK (skipped=1)
2023-01-11T21:40:54.4790999Z 
2023-01-11T21:40:54.4791125Z Generating XML reports...
2023-01-11T21:40:54.4791571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212207.xml
2023-01-11T21:40:54.4791941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4792116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4792497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4792686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4792706Z 
2023-01-11T21:40:54.4792813Z Running tests...
2023-01-11T21:40:54.4793055Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4793370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4793660Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4793881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7689
2023-01-11T21:40:54.4794095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7690
2023-01-11T21:40:54.4794464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4794642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4795020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4795208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4795552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4795726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4796099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4796287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4796532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4796834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4797238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4797633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4797861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4798113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4798280Z skip: Skipped due to small world size. (4.253s)
2023-01-11T21:40:54.4798299Z 
2023-01-11T21:40:54.4798562Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4798673Z Ran 1 test in 4.253s
2023-01-11T21:40:54.4798692Z 
2023-01-11T21:40:54.4798797Z OK (skipped=1)
2023-01-11T21:40:54.4798820Z 
2023-01-11T21:40:54.4798943Z Generating XML reports...
2023-01-11T21:40:54.4799606Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212209.xml
2023-01-11T21:40:54.4799995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4800153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4800535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4800724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4800744Z 
2023-01-11T21:40:54.4800850Z Running tests...
2023-01-11T21:40:54.4801111Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4801422Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4801712Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4801732Z 
2023-01-11T21:40:54.4801988Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4802099Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4802119Z 
2023-01-11T21:40:54.4802207Z OK (skipped=1)
2023-01-11T21:40:54.4802227Z 
2023-01-11T21:40:54.4802350Z Generating XML reports...
2023-01-11T21:40:54.4802795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212216.xml
2023-01-11T21:40:54.4803165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4803339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4803713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4803906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4803926Z 
2023-01-11T21:40:54.4804031Z Running tests...
2023-01-11T21:40:54.4804289Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4804586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4804883Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4804902Z 
2023-01-11T21:40:54.4805162Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4805273Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4805292Z 
2023-01-11T21:40:54.4805397Z OK (skipped=1)
2023-01-11T21:40:54.4805417Z 
2023-01-11T21:40:54.4805538Z Generating XML reports...
2023-01-11T21:40:54.4805985Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212218.xml
2023-01-11T21:40:54.4806440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4806614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4806977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4807167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4807187Z 
2023-01-11T21:40:54.4807294Z Running tests...
2023-01-11T21:40:54.4807620Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4807942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4808222Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4808438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7860
2023-01-11T21:40:54.4808655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7861
2023-01-11T21:40:54.4809008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4809184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4809561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4809753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4810117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4810290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4810665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4810855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4811099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4811326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4811729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4812127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4812355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4812580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4812681Z ok (6.940s)
2023-01-11T21:40:54.4812701Z 
2023-01-11T21:40:54.4812967Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4813125Z Ran 1 test in 6.941s
2023-01-11T21:40:54.4813146Z 
2023-01-11T21:40:54.4813221Z OK
2023-01-11T21:40:54.4813257Z 
2023-01-11T21:40:54.4813365Z Generating XML reports...
2023-01-11T21:40:54.4813816Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212221.xml
2023-01-11T21:40:54.4814182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4814359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4814737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4814929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4814948Z 
2023-01-11T21:40:54.4815056Z Running tests...
2023-01-11T21:40:54.4815314Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4815672Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4815967Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4816182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7977
2023-01-11T21:40:54.4816394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7978
2023-01-11T21:40:54.4816803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4816984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4817361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4817549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4817900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4818073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4818440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4818625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4818871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4819112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4819511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4819907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4820140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4820350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4820453Z ok (6.951s)
2023-01-11T21:40:54.4820473Z 
2023-01-11T21:40:54.4820738Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4820847Z Ran 1 test in 6.951s
2023-01-11T21:40:54.4820867Z 
2023-01-11T21:40:54.4820958Z OK
2023-01-11T21:40:54.4820977Z 
2023-01-11T21:40:54.4821104Z Generating XML reports...
2023-01-11T21:40:54.4821557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212230.xml
2023-01-11T21:40:54.4821924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4822100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4822466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4822656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4822675Z 
2023-01-11T21:40:54.4822781Z Running tests...
2023-01-11T21:40:54.4823040Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4823352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4823657Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4823677Z 
2023-01-11T21:40:54.4823933Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4824043Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4824063Z 
2023-01-11T21:40:54.4824168Z OK (skipped=1)
2023-01-11T21:40:54.4824241Z 
2023-01-11T21:40:54.4824351Z Generating XML reports...
2023-01-11T21:40:54.4824803Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212240.xml
2023-01-11T21:40:54.4825170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4825344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4825760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4825957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4825977Z 
2023-01-11T21:40:54.4826084Z Running tests...
2023-01-11T21:40:54.4826345Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4826641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4826941Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4827156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8127
2023-01-11T21:40:54.4827371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8128
2023-01-11T21:40:54.4827738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4827912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4828294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4828483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4828850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4829008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4829382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4829569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4829812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4830056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4830457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4830854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4831081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4831326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.4831533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4831766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.4832164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4832557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4832658Z ok (7.042s)
2023-01-11T21:40:54.4832677Z 
2023-01-11T21:40:54.4832937Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4833047Z Ran 1 test in 7.043s
2023-01-11T21:40:54.4833066Z 
2023-01-11T21:40:54.4833157Z OK
2023-01-11T21:40:54.4833176Z 
2023-01-11T21:40:54.4833283Z Generating XML reports...
2023-01-11T21:40:54.4833795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212242.xml
2023-01-11T21:40:54.4834163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4834336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4834714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4834946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4834967Z 
2023-01-11T21:40:54.4835079Z Running tests...
2023-01-11T21:40:54.4835339Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4835650Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4835928Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:40:54.4835968Z 
2023-01-11T21:40:54.4836210Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4836321Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4836341Z 
2023-01-11T21:40:54.4836446Z OK (skipped=1)
2023-01-11T21:40:54.4836465Z 
2023-01-11T21:40:54.4836583Z Generating XML reports...
2023-01-11T21:40:54.4837029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212252.xml
2023-01-11T21:40:54.4837403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4837577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4837955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4838127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4838150Z 
2023-01-11T21:40:54.4838257Z Running tests...
2023-01-11T21:40:54.4838514Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4838827Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4839118Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4839331Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8279
2023-01-11T21:40:54.4839761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8280
2023-01-11T21:40:54.4840135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4840293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4840668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4840857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4841216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4841385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4841758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4841944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4842188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4842426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4842811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4843343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4843570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4843797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4843954Z skip: Skipped due to small world size. (4.233s)
2023-01-11T21:40:54.4843974Z 
2023-01-11T21:40:54.4844287Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4844401Z Ran 1 test in 4.233s
2023-01-11T21:40:54.4844421Z 
2023-01-11T21:40:54.4844528Z OK (skipped=1)
2023-01-11T21:40:54.4844547Z 
2023-01-11T21:40:54.4844668Z Generating XML reports...
2023-01-11T21:40:54.4845103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212254.xml
2023-01-11T21:40:54.4845478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4845648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4846026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4846212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4846232Z 
2023-01-11T21:40:54.4846334Z Running tests...
2023-01-11T21:40:54.4846597Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4846907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4847154Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4847371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8384
2023-01-11T21:40:54.4847587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8385
2023-01-11T21:40:54.4847952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4848122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4848495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4848680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4849045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4849216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4849572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4849755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4849997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4850236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4850635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4851030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4851255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4851478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4851713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.4851937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.4852385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4852776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4852873Z ok (6.936s)
2023-01-11T21:40:54.4852893Z 
2023-01-11T21:40:54.4853154Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4853305Z Ran 1 test in 6.936s
2023-01-11T21:40:54.4853326Z 
2023-01-11T21:40:54.4853418Z OK
2023-01-11T21:40:54.4853438Z 
2023-01-11T21:40:54.4853560Z Generating XML reports...
2023-01-11T21:40:54.4853998Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212301.xml
2023-01-11T21:40:54.4854364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4854539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4854913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4855101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4855120Z 
2023-01-11T21:40:54.4855227Z Running tests...
2023-01-11T21:40:54.4855483Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4855799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4856056Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4856255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8508
2023-01-11T21:40:54.4856469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8509
2023-01-11T21:40:54.4856836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4857007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4857382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4857571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4857932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4858103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4858458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4858641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4858883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4859127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4859525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4859915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4860198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4860424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4860570Z skip: Need at least 3 CUDA devices (4.224s)
2023-01-11T21:40:54.4860590Z 
2023-01-11T21:40:54.4860837Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4860945Z Ran 1 test in 4.224s
2023-01-11T21:40:54.4861019Z 
2023-01-11T21:40:54.4861128Z OK (skipped=1)
2023-01-11T21:40:54.4861147Z 
2023-01-11T21:40:54.4861265Z Generating XML reports...
2023-01-11T21:40:54.4861716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212310.xml
2023-01-11T21:40:54.4862080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4862250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4862667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4862858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4862878Z 
2023-01-11T21:40:54.4862970Z Running tests...
2023-01-11T21:40:54.4863231Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4863538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4863790Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2023-01-11T21:40:54.4863810Z 
2023-01-11T21:40:54.4864066Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4864175Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4864195Z 
2023-01-11T21:40:54.4864296Z OK (skipped=1)
2023-01-11T21:40:54.4864315Z 
2023-01-11T21:40:54.4864433Z Generating XML reports...
2023-01-11T21:40:54.4864865Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212317.xml
2023-01-11T21:40:54.4865235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4865405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4865773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4865965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4865985Z 
2023-01-11T21:40:54.4866086Z Running tests...
2023-01-11T21:40:54.4866341Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4866648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4866893Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T21:40:54.4866916Z 
2023-01-11T21:40:54.4867158Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4867263Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4867283Z 
2023-01-11T21:40:54.4867388Z OK (skipped=1)
2023-01-11T21:40:54.4867406Z 
2023-01-11T21:40:54.4867526Z Generating XML reports...
2023-01-11T21:40:54.4867965Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212320.xml
2023-01-11T21:40:54.4868332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4868504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4868876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4869060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4869079Z 
2023-01-11T21:40:54.4869174Z Running tests...
2023-01-11T21:40:54.4869431Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4869736Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4874640Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4874860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8679
2023-01-11T21:40:54.4875175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8680
2023-01-11T21:40:54.4875581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4875756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4876135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4876363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4876734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4876907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4877280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4877470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4877712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4877952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4878348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4878731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4878958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4879185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4879283Z ok (7.623s)
2023-01-11T21:40:54.4879305Z 
2023-01-11T21:40:54.4879938Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4880050Z Ran 1 test in 7.623s
2023-01-11T21:40:54.4880072Z 
2023-01-11T21:40:54.4880161Z OK
2023-01-11T21:40:54.4880181Z 
2023-01-11T21:40:54.4880305Z Generating XML reports...
2023-01-11T21:40:54.4880764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212322.xml
2023-01-11T21:40:54.4881119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4881302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4881676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4881862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4881883Z 
2023-01-11T21:40:54.4881988Z Running tests...
2023-01-11T21:40:54.4882253Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4882564Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4882829Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T21:40:54.4882849Z 
2023-01-11T21:40:54.4883101Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4883196Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4883216Z 
2023-01-11T21:40:54.4883322Z OK (skipped=1)
2023-01-11T21:40:54.4883344Z 
2023-01-11T21:40:54.4883466Z Generating XML reports...
2023-01-11T21:40:54.4883910Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212332.xml
2023-01-11T21:40:54.4884274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4884447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4884926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4885111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4885131Z 
2023-01-11T21:40:54.4885221Z Running tests...
2023-01-11T21:40:54.4885480Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4885787Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4886107Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4886328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8827
2023-01-11T21:40:54.4886540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8828
2023-01-11T21:40:54.4886914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4887090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4887467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4887641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4888007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4888182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4888553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4888738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4888979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4889225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4889624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4890018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4890232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4890461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4890615Z skip: Skipped due to small world size. (4.217s)
2023-01-11T21:40:54.4890635Z 
2023-01-11T21:40:54.4890896Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4891003Z Ran 1 test in 4.217s
2023-01-11T21:40:54.4891023Z 
2023-01-11T21:40:54.4891128Z OK (skipped=1)
2023-01-11T21:40:54.4891151Z 
2023-01-11T21:40:54.4891270Z Generating XML reports...
2023-01-11T21:40:54.4891717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212335.xml
2023-01-11T21:40:54.4892071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4892248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4892625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4892813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4892833Z 
2023-01-11T21:40:54.4892938Z Running tests...
2023-01-11T21:40:54.4893191Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4893501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4893818Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T21:40:54.4893838Z 
2023-01-11T21:40:54.4894095Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4894189Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4894209Z 
2023-01-11T21:40:54.4894314Z OK (skipped=1)
2023-01-11T21:40:54.4894333Z 
2023-01-11T21:40:54.4894454Z Generating XML reports...
2023-01-11T21:40:54.4894945Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212341.xml
2023-01-11T21:40:54.4895323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4895495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4895873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4896066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4896085Z 
2023-01-11T21:40:54.4896188Z Running tests...
2023-01-11T21:40:54.4896433Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4896743Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4897000Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4897221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8965
2023-01-11T21:40:54.4897434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8966
2023-01-11T21:40:54.4897800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4897973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4898351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4898526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4898887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4899057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4899421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4899606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4899845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4900086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4900478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4900874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4901087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4901311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4901467Z skip: Skipped due to small world size. (4.231s)
2023-01-11T21:40:54.4901487Z 
2023-01-11T21:40:54.4901750Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4901860Z Ran 1 test in 4.232s
2023-01-11T21:40:54.4901879Z 
2023-01-11T21:40:54.4901982Z OK (skipped=1)
2023-01-11T21:40:54.4902001Z 
2023-01-11T21:40:54.4902123Z Generating XML reports...
2023-01-11T21:40:54.4902568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212344.xml
2023-01-11T21:40:54.4902991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4903166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4903538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4903727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4903747Z 
2023-01-11T21:40:54.4903854Z Running tests...
2023-01-11T21:40:54.4904150Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4904470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4904746Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T21:40:54.4904766Z 
2023-01-11T21:40:54.4905023Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4905121Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4905156Z 
2023-01-11T21:40:54.4905246Z OK (skipped=1)
2023-01-11T21:40:54.4905265Z 
2023-01-11T21:40:54.4905387Z Generating XML reports...
2023-01-11T21:40:54.4905830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212351.xml
2023-01-11T21:40:54.4906198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4906375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4906749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4906936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4906955Z 
2023-01-11T21:40:54.4907060Z Running tests...
2023-01-11T21:40:54.4907303Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4907617Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4907889Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T21:40:54.4907909Z 
2023-01-11T21:40:54.4908163Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4908270Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4908289Z 
2023-01-11T21:40:54.4908392Z OK (skipped=1)
2023-01-11T21:40:54.4908415Z 
2023-01-11T21:40:54.4908533Z Generating XML reports...
2023-01-11T21:40:54.4908980Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212353.xml
2023-01-11T21:40:54.4909346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4909504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4909883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4910070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4910090Z 
2023-01-11T21:40:54.4910194Z Running tests...
2023-01-11T21:40:54.4910447Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4910754Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4911027Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T21:40:54.4911048Z 
2023-01-11T21:40:54.4911300Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4911394Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4911429Z 
2023-01-11T21:40:54.4911518Z OK (skipped=1)
2023-01-11T21:40:54.4911537Z 
2023-01-11T21:40:54.4911713Z Generating XML reports...
2023-01-11T21:40:54.4912163Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212355.xml
2023-01-11T21:40:54.4912526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4912697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4913116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4913350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4913371Z 
2023-01-11T21:40:54.4913479Z Running tests...
2023-01-11T21:40:54.4913727Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4914037Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4914293Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T21:40:54.4914316Z 
2023-01-11T21:40:54.4914571Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4914681Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4914701Z 
2023-01-11T21:40:54.4914806Z OK (skipped=1)
2023-01-11T21:40:54.4914825Z 
2023-01-11T21:40:54.4914948Z Generating XML reports...
2023-01-11T21:40:54.4915396Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212358.xml
2023-01-11T21:40:54.4915765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4915924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4916300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4916492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4916512Z 
2023-01-11T21:40:54.4916620Z Running tests...
2023-01-11T21:40:54.4916876Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4917184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4917445Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T21:40:54.4917465Z 
2023-01-11T21:40:54.4917721Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4917815Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4917852Z 
2023-01-11T21:40:54.4917941Z OK (skipped=1)
2023-01-11T21:40:54.4917960Z 
2023-01-11T21:40:54.4918080Z Generating XML reports...
2023-01-11T21:40:54.4918530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212400.xml
2023-01-11T21:40:54.4918905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4919078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4919684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4919886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4919907Z 
2023-01-11T21:40:54.4920018Z Running tests...
2023-01-11T21:40:54.4920270Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4920584Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4920865Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4921082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9235
2023-01-11T21:40:54.4921383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9236
2023-01-11T21:40:54.4921758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4921933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4922306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4922494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4922898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4923078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4923459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4923644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4923892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4924136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4924534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4924932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4925145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4925370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4925609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.4925845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.4926241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4926632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.4926872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.4927112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.4927503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.4927893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.4927978Z ok (4.416s)
2023-01-11T21:40:54.4928001Z 
2023-01-11T21:40:54.4928264Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4928374Z Ran 1 test in 4.416s
2023-01-11T21:40:54.4928394Z 
2023-01-11T21:40:54.4928485Z OK
2023-01-11T21:40:54.4928504Z 
2023-01-11T21:40:54.4928627Z Generating XML reports...
2023-01-11T21:40:54.4929075Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212402.xml
2023-01-11T21:40:54.4929448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4929623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4929987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4930178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4930197Z 
2023-01-11T21:40:54.4930303Z Running tests...
2023-01-11T21:40:54.4930624Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4930936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4931198Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4931416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9348
2023-01-11T21:40:54.4931626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9349
2023-01-11T21:40:54.4932051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4932217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4932599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4932791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4933158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4933330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4933700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4933888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4934136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4934361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4934765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4935158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4935389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4935613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4935714Z ok (5.826s)
2023-01-11T21:40:54.4935734Z 
2023-01-11T21:40:54.4935999Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4936111Z Ran 1 test in 5.827s
2023-01-11T21:40:54.4936131Z 
2023-01-11T21:40:54.4936223Z OK
2023-01-11T21:40:54.4936245Z 
2023-01-11T21:40:54.4936352Z Generating XML reports...
2023-01-11T21:40:54.4936801Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212409.xml
2023-01-11T21:40:54.4937172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4937346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4937731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4937921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4937941Z 
2023-01-11T21:40:54.4938048Z Running tests...
2023-01-11T21:40:54.4938307Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4938601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4938885Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4939100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9464
2023-01-11T21:40:54.4939310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9465
2023-01-11T21:40:54.4939680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4939912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4940295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4940485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4940854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4941053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4941434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4941621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4941867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4942112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4942509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4942900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4943131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4943363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4943505Z skip: Skipped due to small world size. (4.326s)
2023-01-11T21:40:54.4943525Z 
2023-01-11T21:40:54.4943787Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4943898Z Ran 1 test in 4.326s
2023-01-11T21:40:54.4943917Z 
2023-01-11T21:40:54.4944023Z OK (skipped=1)
2023-01-11T21:40:54.4944046Z 
2023-01-11T21:40:54.4944171Z Generating XML reports...
2023-01-11T21:40:54.4944621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212418.xml
2023-01-11T21:40:54.4944991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4945168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4945533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4945723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4945743Z 
2023-01-11T21:40:54.4945850Z Running tests...
2023-01-11T21:40:54.4946112Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4946421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4946692Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4946909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9569
2023-01-11T21:40:54.4947123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9570
2023-01-11T21:40:54.4947491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4947649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4948026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4948214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4948579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4948804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4949185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4949372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4949615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4949840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4950280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4950683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4950914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4951146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4951247Z ok (5.642s)
2023-01-11T21:40:54.4951267Z 
2023-01-11T21:40:54.4951530Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4951642Z Ran 1 test in 5.642s
2023-01-11T21:40:54.4951661Z 
2023-01-11T21:40:54.4951752Z OK
2023-01-11T21:40:54.4951771Z 
2023-01-11T21:40:54.4951877Z Generating XML reports...
2023-01-11T21:40:54.4952326Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212425.xml
2023-01-11T21:40:54.4952693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4952867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4953245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4953440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4953459Z 
2023-01-11T21:40:54.4953568Z Running tests...
2023-01-11T21:40:54.4953827Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4954138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4954394Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4954613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9675
2023-01-11T21:40:54.4954827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9676
2023-01-11T21:40:54.4955197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4955371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4955746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4955937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4956297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4956454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4956823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4957010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4957254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4957495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4957893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4958349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4958578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4958806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4958890Z ok (4.335s)
2023-01-11T21:40:54.4958911Z 
2023-01-11T21:40:54.4959226Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4959347Z Ran 1 test in 4.335s
2023-01-11T21:40:54.4959567Z 
2023-01-11T21:40:54.4959677Z OK
2023-01-11T21:40:54.4959697Z 
2023-01-11T21:40:54.4959825Z Generating XML reports...
2023-01-11T21:40:54.4960291Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212433.xml
2023-01-11T21:40:54.4960662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4960844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4961205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4961393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4961413Z 
2023-01-11T21:40:54.4961520Z Running tests...
2023-01-11T21:40:54.4961782Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4962093Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4962375Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4962591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9780
2023-01-11T21:40:54.4962811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9781
2023-01-11T21:40:54.4963181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4963339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4963717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4963905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4964270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4964445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4964815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4965001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4965249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4965475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4965879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4966271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4966500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4966726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4966827Z ok (5.851s)
2023-01-11T21:40:54.4966847Z 
2023-01-11T21:40:54.4967110Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4967306Z Ran 1 test in 5.851s
2023-01-11T21:40:54.4967326Z 
2023-01-11T21:40:54.4967419Z OK
2023-01-11T21:40:54.4967438Z 
2023-01-11T21:40:54.4967544Z Generating XML reports...
2023-01-11T21:40:54.4967998Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212440.xml
2023-01-11T21:40:54.4968365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4968542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4968973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4969173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4969193Z 
2023-01-11T21:40:54.4969302Z Running tests...
2023-01-11T21:40:54.4969563Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4969881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4970135Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4970354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9896
2023-01-11T21:40:54.4970565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9897
2023-01-11T21:40:54.4970937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4971110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4971490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4971678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4972037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4972195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4972566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4972755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4972998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4973243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4973640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4974034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4974261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4974492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4974576Z ok (7.031s)
2023-01-11T21:40:54.4974596Z 
2023-01-11T21:40:54.4974858Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4974969Z Ran 1 test in 7.031s
2023-01-11T21:40:54.4974989Z 
2023-01-11T21:40:54.4975079Z OK
2023-01-11T21:40:54.4975098Z 
2023-01-11T21:40:54.4975222Z Generating XML reports...
2023-01-11T21:40:54.4975673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212448.xml
2023-01-11T21:40:54.4976044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4976216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4976578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4976826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4976846Z 
2023-01-11T21:40:54.4976955Z Running tests...
2023-01-11T21:40:54.4977216Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4977528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4977885Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4978115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10011
2023-01-11T21:40:54.4978334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10012
2023-01-11T21:40:54.4978707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4978866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4979244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4979432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4979795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4979968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4980341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4980526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4980774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.4981000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.4981405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4981799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.4982027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.4982255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.4982359Z ok (4.338s)
2023-01-11T21:40:54.4982381Z 
2023-01-11T21:40:54.4982646Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4982759Z Ran 1 test in 4.338s
2023-01-11T21:40:54.4982779Z 
2023-01-11T21:40:54.4982870Z OK
2023-01-11T21:40:54.4982889Z 
2023-01-11T21:40:54.4982995Z Generating XML reports...
2023-01-11T21:40:54.4983444Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212458.xml
2023-01-11T21:40:54.4983817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4983991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4984370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4984562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4984582Z 
2023-01-11T21:40:54.4984691Z Running tests...
2023-01-11T21:40:54.4984951Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4985262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4985500Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4985573Z 
2023-01-11T21:40:54.4985836Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4985948Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4985968Z 
2023-01-11T21:40:54.4986076Z OK (skipped=1)
2023-01-11T21:40:54.4986094Z 
2023-01-11T21:40:54.4986218Z Generating XML reports...
2023-01-11T21:40:54.4986662Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212505.xml
2023-01-11T21:40:54.4987072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4987254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4987617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4987806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4987826Z 
2023-01-11T21:40:54.4987935Z Running tests...
2023-01-11T21:40:54.4988201Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4988511Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4988763Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.4989521Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.610s)
2023-01-11T21:40:54.4989543Z 
2023-01-11T21:40:54.4989804Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4989917Z Ran 1 test in 1.610s
2023-01-11T21:40:54.4989937Z 
2023-01-11T21:40:54.4990045Z OK (skipped=1)
2023-01-11T21:40:54.4990064Z 
2023-01-11T21:40:54.4990170Z Generating XML reports...
2023-01-11T21:40:54.4990621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212507.xml
2023-01-11T21:40:54.4990987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4991160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4991538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4991731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4991751Z 
2023-01-11T21:40:54.4991858Z Running tests...
2023-01-11T21:40:54.4992117Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4992413Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4992681Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4992706Z 
2023-01-11T21:40:54.4992969Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4993078Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4993097Z 
2023-01-11T21:40:54.4993203Z OK (skipped=1)
2023-01-11T21:40:54.4993221Z 
2023-01-11T21:40:54.4993344Z Generating XML reports...
2023-01-11T21:40:54.4993789Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212511.xml
2023-01-11T21:40:54.4994160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4994335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4994697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4994887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4994958Z 
2023-01-11T21:40:54.4995071Z Running tests...
2023-01-11T21:40:54.4995333Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4995646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4995907Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.4995926Z 
2023-01-11T21:40:54.4996182Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4996349Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4996370Z 
2023-01-11T21:40:54.4996483Z OK (skipped=1)
2023-01-11T21:40:54.4996502Z 
2023-01-11T21:40:54.4996607Z Generating XML reports...
2023-01-11T21:40:54.4997055Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212514.xml
2023-01-11T21:40:54.4997422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.4997600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.4997974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.4998162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.4998182Z 
2023-01-11T21:40:54.4998289Z Running tests...
2023-01-11T21:40:54.4998550Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4998864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.4999116Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2023-01-11T21:40:54.4999136Z 
2023-01-11T21:40:54.4999610Z ----------------------------------------------------------------------
2023-01-11T21:40:54.4999739Z Ran 1 test in 0.002s
2023-01-11T21:40:54.4999759Z 
2023-01-11T21:40:54.4999868Z OK (skipped=1)
2023-01-11T21:40:54.4999887Z 
2023-01-11T21:40:54.5000012Z Generating XML reports...
2023-01-11T21:40:54.5000468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212516.xml
2023-01-11T21:40:54.5000837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5001014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5001396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5001569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5001588Z 
2023-01-11T21:40:54.5001696Z Running tests...
2023-01-11T21:40:54.5001956Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5002267Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5002528Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5003275Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2023-01-11T21:40:54.5003296Z 
2023-01-11T21:40:54.5003559Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5003673Z Ran 1 test in 1.630s
2023-01-11T21:40:54.5003692Z 
2023-01-11T21:40:54.5003797Z OK (skipped=1)
2023-01-11T21:40:54.5003816Z 
2023-01-11T21:40:54.5003922Z Generating XML reports...
2023-01-11T21:40:54.5004365Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212518.xml
2023-01-11T21:40:54.5004825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5005002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5005378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5005570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5005589Z 
2023-01-11T21:40:54.5005695Z Running tests...
2023-01-11T21:40:54.5006010Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5006336Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5006634Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5007385Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.596s)
2023-01-11T21:40:54.5007405Z 
2023-01-11T21:40:54.5007664Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5007777Z Ran 1 test in 1.596s
2023-01-11T21:40:54.5007797Z 
2023-01-11T21:40:54.5007904Z OK (skipped=1)
2023-01-11T21:40:54.5007923Z 
2023-01-11T21:40:54.5008050Z Generating XML reports...
2023-01-11T21:40:54.5008500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212522.xml
2023-01-11T21:40:54.5008867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5009043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5009425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5009598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5009617Z 
2023-01-11T21:40:54.5009725Z Running tests...
2023-01-11T21:40:54.5009981Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5010291Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5010609Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5011357Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2023-01-11T21:40:54.5011381Z 
2023-01-11T21:40:54.5011638Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5011751Z Ran 1 test in 1.613s
2023-01-11T21:40:54.5011771Z 
2023-01-11T21:40:54.5011880Z OK (skipped=1)
2023-01-11T21:40:54.5011899Z 
2023-01-11T21:40:54.5012005Z Generating XML reports...
2023-01-11T21:40:54.5012454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212527.xml
2023-01-11T21:40:54.5012823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5013038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5013421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5013612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5013684Z 
2023-01-11T21:40:54.5013799Z Running tests...
2023-01-11T21:40:54.5014063Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5014378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5014636Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5014855Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10384
2023-01-11T21:40:54.5015115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10385
2023-01-11T21:40:54.5015495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5015671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5016051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5016245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5016612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5016770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5017147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5017339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5017584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5017830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5018230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5018628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5018857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5019085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5019869Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:40:54.5019965Z   warnings.warn(
2023-01-11T21:40:54.5020739Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:40:54.5020853Z   warnings.warn(
2023-01-11T21:40:54.5021087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5021318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5021552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5021787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5022020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5022247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5022452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5022738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5022841Z ok (7.741s)
2023-01-11T21:40:54.5022861Z 
2023-01-11T21:40:54.5023129Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5023243Z Ran 1 test in 7.742s
2023-01-11T21:40:54.5023263Z 
2023-01-11T21:40:54.5023356Z OK
2023-01-11T21:40:54.5023375Z 
2023-01-11T21:40:54.5023500Z Generating XML reports...
2023-01-11T21:40:54.5023994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212531.xml
2023-01-11T21:40:54.5024354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5024532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5024914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5025109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5025128Z 
2023-01-11T21:40:54.5025235Z Running tests...
2023-01-11T21:40:54.5025494Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5025807Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5026118Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5026340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10502
2023-01-11T21:40:54.5026541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10503
2023-01-11T21:40:54.5026911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5027084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5027465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5027652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5028012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5028183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5028554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5028724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5028970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5029212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5029610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5030011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5030242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5030469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5031250Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:40:54.5031364Z   warnings.warn(
2023-01-11T21:40:54.5032137Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:40:54.5032305Z   warnings.warn(
2023-01-11T21:40:54.5032523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5032751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5033023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5033263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5033366Z ok (6.853s)
2023-01-11T21:40:54.5033386Z 
2023-01-11T21:40:54.5033654Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5033766Z Ran 1 test in 6.853s
2023-01-11T21:40:54.5033786Z 
2023-01-11T21:40:54.5033883Z OK
2023-01-11T21:40:54.5033902Z 
2023-01-11T21:40:54.5034008Z Generating XML reports...
2023-01-11T21:40:54.5034458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212541.xml
2023-01-11T21:40:54.5034827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5035001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5035383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5035574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5035594Z 
2023-01-11T21:40:54.5035701Z Running tests...
2023-01-11T21:40:54.5035961Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5036255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5036555Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5036771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10620
2023-01-11T21:40:54.5036984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10621
2023-01-11T21:40:54.5037353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5037530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5037910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5038097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5038464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5038622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5038992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5039176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5039636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5039890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5040303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5040702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5040930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5041242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5042011Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:40:54.5042126Z   warnings.warn(
2023-01-11T21:40:54.5042956Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:40:54.5043074Z   warnings.warn(
2023-01-11T21:40:54.5043176Z ok (7.557s)
2023-01-11T21:40:54.5043197Z 
2023-01-11T21:40:54.5043461Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5043576Z Ran 1 test in 7.557s
2023-01-11T21:40:54.5043596Z 
2023-01-11T21:40:54.5043687Z OK
2023-01-11T21:40:54.5043706Z 
2023-01-11T21:40:54.5043830Z Generating XML reports...
2023-01-11T21:40:54.5044263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212550.xml
2023-01-11T21:40:54.5044632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5044809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5045187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5045377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5045397Z 
2023-01-11T21:40:54.5045504Z Running tests...
2023-01-11T21:40:54.5045762Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5046078Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5046341Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5046543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10740
2023-01-11T21:40:54.5046758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10741
2023-01-11T21:40:54.5047134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5047308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5047687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5047876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5048246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5048416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5048789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5048958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5049205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5049448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5049846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5050240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5050527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5050756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5050991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5051207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5051309Z ok (6.350s)
2023-01-11T21:40:54.5051330Z 
2023-01-11T21:40:54.5051639Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5051759Z Ran 1 test in 6.350s
2023-01-11T21:40:54.5051779Z 
2023-01-11T21:40:54.5051871Z OK
2023-01-11T21:40:54.5051890Z 
2023-01-11T21:40:54.5052014Z Generating XML reports...
2023-01-11T21:40:54.5052466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212601.xml
2023-01-11T21:40:54.5052841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5053017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5053377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5053566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5053586Z 
2023-01-11T21:40:54.5053692Z Running tests...
2023-01-11T21:40:54.5053956Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5054267Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5054542Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5054762Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10858
2023-01-11T21:40:54.5054982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10859
2023-01-11T21:40:54.5055335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5055510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5055887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5056075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5056440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5056614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5056985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5057174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5057419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5057803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5058044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5058438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5058666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5058892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5059126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5059358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5059643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5059877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5059961Z ok (6.231s)
2023-01-11T21:40:54.5059981Z 
2023-01-11T21:40:54.5060248Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5060358Z Ran 1 test in 6.231s
2023-01-11T21:40:54.5060378Z 
2023-01-11T21:40:54.5060470Z OK
2023-01-11T21:40:54.5060544Z 
2023-01-11T21:40:54.5060675Z Generating XML reports...
2023-01-11T21:40:54.5061128Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212609.xml
2023-01-11T21:40:54.5061497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5061678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5062036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5062224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5062244Z 
2023-01-11T21:40:54.5062350Z Running tests...
2023-01-11T21:40:54.5062611Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5062919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5063193Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5063941Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2023-01-11T21:40:54.5063965Z 
2023-01-11T21:40:54.5064229Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5064343Z Ran 1 test in 1.621s
2023-01-11T21:40:54.5064363Z 
2023-01-11T21:40:54.5064471Z OK (skipped=1)
2023-01-11T21:40:54.5064490Z 
2023-01-11T21:40:54.5064595Z Generating XML reports...
2023-01-11T21:40:54.5065038Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212618.xml
2023-01-11T21:40:54.5065417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5065592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5065968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5066157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5066180Z 
2023-01-11T21:40:54.5066288Z Running tests...
2023-01-11T21:40:54.5066548Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5066843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5067133Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5067881Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2023-01-11T21:40:54.5067903Z 
2023-01-11T21:40:54.5068161Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5068273Z Ran 1 test in 1.627s
2023-01-11T21:40:54.5068292Z 
2023-01-11T21:40:54.5068454Z OK (skipped=1)
2023-01-11T21:40:54.5068474Z 
2023-01-11T21:40:54.5068598Z Generating XML reports...
2023-01-11T21:40:54.5069045Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212622.xml
2023-01-11T21:40:54.5069413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5069588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5069990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5070188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5070208Z 
2023-01-11T21:40:54.5070317Z Running tests...
2023-01-11T21:40:54.5070578Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5070890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5071180Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5071401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11044
2023-01-11T21:40:54.5071618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11045
2023-01-11T21:40:54.5071968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5072147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5072527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5072716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5073076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5073249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5073621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5073807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5074053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5074282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5074684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5075079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5075308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5075539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5075748Z <class 'torch.nn.parameter.Parameter'>
2023-01-11T21:40:54.5075849Z ok (6.841s)
2023-01-11T21:40:54.5075870Z 
2023-01-11T21:40:54.5076133Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5076227Z Ran 1 test in 6.841s
2023-01-11T21:40:54.5076265Z 
2023-01-11T21:40:54.5076340Z OK
2023-01-11T21:40:54.5076359Z 
2023-01-11T21:40:54.5076482Z Generating XML reports...
2023-01-11T21:40:54.5076935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212627.xml
2023-01-11T21:40:54.5077305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5077481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5077861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5078110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5078130Z 
2023-01-11T21:40:54.5078240Z Running tests...
2023-01-11T21:40:54.5078484Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5078798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5079152Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5079643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11159
2023-01-11T21:40:54.5079875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11160
2023-01-11T21:40:54.5080255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5080437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5080816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5081006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5081352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5081521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5081901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5082086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5082336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5082576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5082979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5083372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5083584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5083814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5083922Z ok (6.830s)
2023-01-11T21:40:54.5083943Z 
2023-01-11T21:40:54.5084207Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5084322Z Ran 1 test in 6.831s
2023-01-11T21:40:54.5084341Z 
2023-01-11T21:40:54.5084432Z OK
2023-01-11T21:40:54.5084451Z 
2023-01-11T21:40:54.5084574Z Generating XML reports...
2023-01-11T21:40:54.5085025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212636.xml
2023-01-11T21:40:54.5085401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5085559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5085936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5086124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5086147Z 
2023-01-11T21:40:54.5086254Z Running tests...
2023-01-11T21:40:54.5086512Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5086823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5087086Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5087390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11274
2023-01-11T21:40:54.5087588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11275
2023-01-11T21:40:54.5087964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5088138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5088515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5088762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5089136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5089309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5089682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5089870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5090098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5090340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5090739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5091133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5091363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5091588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5091823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5092053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5092138Z ok (7.328s)
2023-01-11T21:40:54.5092176Z 
2023-01-11T21:40:54.5092422Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5092532Z Ran 1 test in 7.328s
2023-01-11T21:40:54.5092552Z 
2023-01-11T21:40:54.5092643Z OK
2023-01-11T21:40:54.5092662Z 
2023-01-11T21:40:54.5092785Z Generating XML reports...
2023-01-11T21:40:54.5093238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212645.xml
2023-01-11T21:40:54.5093612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5093788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5094168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5094346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5094384Z 
2023-01-11T21:40:54.5094475Z Running tests...
2023-01-11T21:40:54.5094735Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5095048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5095339Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5095559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11393
2023-01-11T21:40:54.5095774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11394
2023-01-11T21:40:54.5096147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5096321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5096742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5096932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5097293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5097463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5097875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5098071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5098319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5098566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5098957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5099353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5099582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5099810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5100604Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:40:54.5100711Z ok (6.336s)
2023-01-11T21:40:54.5100731Z 
2023-01-11T21:40:54.5100995Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5101106Z Ran 1 test in 6.336s
2023-01-11T21:40:54.5101125Z 
2023-01-11T21:40:54.5101217Z OK
2023-01-11T21:40:54.5101236Z 
2023-01-11T21:40:54.5101361Z Generating XML reports...
2023-01-11T21:40:54.5101814Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212655.xml
2023-01-11T21:40:54.5102169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5102343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5102724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5102917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5102937Z 
2023-01-11T21:40:54.5103045Z Running tests...
2023-01-11T21:40:54.5103305Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5103618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5103899Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5104653Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2023-01-11T21:40:54.5104674Z 
2023-01-11T21:40:54.5104918Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5105087Z Ran 1 test in 1.621s
2023-01-11T21:40:54.5105107Z 
2023-01-11T21:40:54.5105216Z OK (skipped=1)
2023-01-11T21:40:54.5105235Z 
2023-01-11T21:40:54.5105359Z Generating XML reports...
2023-01-11T21:40:54.5105809Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212704.xml
2023-01-11T21:40:54.5106179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5106356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5106775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5106975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5106994Z 
2023-01-11T21:40:54.5107086Z Running tests...
2023-01-11T21:40:54.5107349Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5107666Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5107992Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2023-01-11T21:40:54.5108012Z 
2023-01-11T21:40:54.5108266Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5108376Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5108396Z 
2023-01-11T21:40:54.5108502Z OK (skipped=1)
2023-01-11T21:40:54.5108521Z 
2023-01-11T21:40:54.5108642Z Generating XML reports...
2023-01-11T21:40:54.5109092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212708.xml
2023-01-11T21:40:54.5109446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5109620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5109998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5110190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5110210Z 
2023-01-11T21:40:54.5110316Z Running tests...
2023-01-11T21:40:54.5110574Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5110884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5111134Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5111879Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.632s)
2023-01-11T21:40:54.5111900Z 
2023-01-11T21:40:54.5112145Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5112262Z Ran 1 test in 1.632s
2023-01-11T21:40:54.5112281Z 
2023-01-11T21:40:54.5112388Z OK (skipped=1)
2023-01-11T21:40:54.5112407Z 
2023-01-11T21:40:54.5112531Z Generating XML reports...
2023-01-11T21:40:54.5112977Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212711.xml
2023-01-11T21:40:54.5113392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5113572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5113950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5114139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5114159Z 
2023-01-11T21:40:54.5114250Z Running tests...
2023-01-11T21:40:54.5114509Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5114880Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5115150Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5115373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11612
2023-01-11T21:40:54.5115589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11613
2023-01-11T21:40:54.5116000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5116182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5116544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5116733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5117098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5117269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5117641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5117829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5118080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5118326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5118726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5119104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5119337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5119794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5120600Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T21:40:54.5120931Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T21:40:54.5121718Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T21:40:54.5122047Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T21:40:54.5122149Z ok (7.458s)
2023-01-11T21:40:54.5122169Z 
2023-01-11T21:40:54.5122430Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5122543Z Ran 1 test in 7.458s
2023-01-11T21:40:54.5122562Z 
2023-01-11T21:40:54.5122655Z OK
2023-01-11T21:40:54.5122674Z 
2023-01-11T21:40:54.5122780Z Generating XML reports...
2023-01-11T21:40:54.5123235Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212715.xml
2023-01-11T21:40:54.5123605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5123778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5124276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5124468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5124488Z 
2023-01-11T21:40:54.5124598Z Running tests...
2023-01-11T21:40:54.5124862Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5125154Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5125498Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5126261Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2023-01-11T21:40:54.5126286Z 
2023-01-11T21:40:54.5126547Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5126659Z Ran 1 test in 1.612s
2023-01-11T21:40:54.5126679Z 
2023-01-11T21:40:54.5126787Z OK (skipped=1)
2023-01-11T21:40:54.5126806Z 
2023-01-11T21:40:54.5126931Z Generating XML reports...
2023-01-11T21:40:54.5127376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212725.xml
2023-01-11T21:40:54.5127745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5127920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5128279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5128467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5128487Z 
2023-01-11T21:40:54.5128593Z Running tests...
2023-01-11T21:40:54.5128856Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5129167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5129436Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5130181Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T21:40:54.5130202Z 
2023-01-11T21:40:54.5130460Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5130570Z Ran 1 test in 1.633s
2023-01-11T21:40:54.5130590Z 
2023-01-11T21:40:54.5130679Z OK (skipped=1)
2023-01-11T21:40:54.5130716Z 
2023-01-11T21:40:54.5130823Z Generating XML reports...
2023-01-11T21:40:54.5131275Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212729.xml
2023-01-11T21:40:54.5131643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5131818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5132202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5132395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5132415Z 
2023-01-11T21:40:54.5132523Z Running tests...
2023-01-11T21:40:54.5132786Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5133083Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5133374Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5133657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11829
2023-01-11T21:40:54.5133873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11830
2023-01-11T21:40:54.5134243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5134420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5134841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5135040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5135406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5135560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5135938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5136126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5136371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5136617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5137019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5137416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5137644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5137870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5138098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5138333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5138732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5139121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5139358Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5139593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5139823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5140054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5140142Z ok (7.745s)
2023-01-11T21:40:54.5140162Z 
2023-01-11T21:40:54.5140427Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5140538Z Ran 1 test in 7.745s
2023-01-11T21:40:54.5140558Z 
2023-01-11T21:40:54.5140650Z OK
2023-01-11T21:40:54.5140670Z 
2023-01-11T21:40:54.5140792Z Generating XML reports...
2023-01-11T21:40:54.5141243Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212733.xml
2023-01-11T21:40:54.5141619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5141795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5142174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5142348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5142419Z 
2023-01-11T21:40:54.5142533Z Running tests...
2023-01-11T21:40:54.5142801Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5143113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5143385Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5143603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11950
2023-01-11T21:40:54.5143868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11951
2023-01-11T21:40:54.5144248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5144405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5144782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5144973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5145333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5145507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5145879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5146072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5146319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5146565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5146949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5147347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5147574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5147850Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:40:54.5148074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5148344Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:40:54.5148581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5148812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5149044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5149263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5149540Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:40:54.5149815Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:40:54.5150086Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:40:54.5150355Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:40:54.5150588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5150820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5151047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5151313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5151590Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:40:54.5151860Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:40:54.5152134Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T21:40:54.5152439Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T21:40:54.5152680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5152912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5153143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5153381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5153464Z ok (8.103s)
2023-01-11T21:40:54.5153500Z 
2023-01-11T21:40:54.5153758Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5153870Z Ran 1 test in 8.103s
2023-01-11T21:40:54.5153890Z 
2023-01-11T21:40:54.5153982Z OK
2023-01-11T21:40:54.5154002Z 
2023-01-11T21:40:54.5154125Z Generating XML reports...
2023-01-11T21:40:54.5154578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212743.xml
2023-01-11T21:40:54.5154950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5155127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5155510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5155688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5155708Z 
2023-01-11T21:40:54.5155816Z Running tests...
2023-01-11T21:40:54.5156074Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5156384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5156655Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5157398Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2023-01-11T21:40:54.5157419Z 
2023-01-11T21:40:54.5157681Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5157798Z Ran 1 test in 1.607s
2023-01-11T21:40:54.5157818Z 
2023-01-11T21:40:54.5157925Z OK (skipped=1)
2023-01-11T21:40:54.5157944Z 
2023-01-11T21:40:54.5158050Z Generating XML reports...
2023-01-11T21:40:54.5158497Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212754.xml
2023-01-11T21:40:54.5158863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5159038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5159628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5159830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5159851Z 
2023-01-11T21:40:54.5159959Z Running tests...
2023-01-11T21:40:54.5160228Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5160627Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5160884Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5161106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12103
2023-01-11T21:40:54.5161321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12104
2023-01-11T21:40:54.5161751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5161935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5162317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5162509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5162875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5163030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5163406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5163594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5163843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5164089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5164490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5164883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5165118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5165667Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:40:54.5165898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5166434Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:40:54.5166657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5166891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5167164Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T21:40:54.5167435Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T21:40:54.5167734Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T21:40:54.5168031Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T21:40:54.5168359Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T21:40:54.5168687Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T21:40:54.5169075Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T21:40:54.5169398Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T21:40:54.5169657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5169898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5169999Z ok (7.405s)
2023-01-11T21:40:54.5170021Z 
2023-01-11T21:40:54.5170293Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5170406Z Ran 1 test in 7.405s
2023-01-11T21:40:54.5170426Z 
2023-01-11T21:40:54.5170522Z OK
2023-01-11T21:40:54.5170542Z 
2023-01-11T21:40:54.5170667Z Generating XML reports...
2023-01-11T21:40:54.5171123Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212758.xml
2023-01-11T21:40:54.5171494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5171654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5172037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5172227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5172247Z 
2023-01-11T21:40:54.5172355Z Running tests...
2023-01-11T21:40:54.5172615Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5172925Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5173323Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5173393Z 
2023-01-11T21:40:54.5173657Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5173771Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5173790Z 
2023-01-11T21:40:54.5173880Z OK (skipped=1)
2023-01-11T21:40:54.5173899Z 
2023-01-11T21:40:54.5174023Z Generating XML reports...
2023-01-11T21:40:54.5174477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212808.xml
2023-01-11T21:40:54.5174847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5175021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5175397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5175589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5175610Z 
2023-01-11T21:40:54.5175716Z Running tests...
2023-01-11T21:40:54.5175961Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5176275Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5176669Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5176690Z 
2023-01-11T21:40:54.5176948Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5177058Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5177078Z 
2023-01-11T21:40:54.5177184Z OK (skipped=1)
2023-01-11T21:40:54.5177203Z 
2023-01-11T21:40:54.5177389Z Generating XML reports...
2023-01-11T21:40:54.5177841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212810.xml
2023-01-11T21:40:54.5178209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5178384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5178745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5178982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5179004Z 
2023-01-11T21:40:54.5179117Z Running tests...
2023-01-11T21:40:54.5179379Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5179691Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5180143Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5180167Z 
2023-01-11T21:40:54.5180427Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5180537Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5180557Z 
2023-01-11T21:40:54.5180664Z OK (skipped=1)
2023-01-11T21:40:54.5180684Z 
2023-01-11T21:40:54.5180789Z Generating XML reports...
2023-01-11T21:40:54.5181241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212813.xml
2023-01-11T21:40:54.5181610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5181784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5182160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5182356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5182375Z 
2023-01-11T21:40:54.5182485Z Running tests...
2023-01-11T21:40:54.5182746Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5183037Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5183487Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5183526Z 
2023-01-11T21:40:54.5183768Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5183879Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5183899Z 
2023-01-11T21:40:54.5184003Z OK (skipped=1)
2023-01-11T21:40:54.5184026Z 
2023-01-11T21:40:54.5184149Z Generating XML reports...
2023-01-11T21:40:54.5184593Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212815.xml
2023-01-11T21:40:54.5184960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5185134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5185512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5185687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5185706Z 
2023-01-11T21:40:54.5185815Z Running tests...
2023-01-11T21:40:54.5186076Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5186383Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5186892Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5186914Z 
2023-01-11T21:40:54.5187174Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5187288Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5187307Z 
2023-01-11T21:40:54.5187415Z OK (skipped=1)
2023-01-11T21:40:54.5187434Z 
2023-01-11T21:40:54.5187612Z Generating XML reports...
2023-01-11T21:40:54.5188051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212818.xml
2023-01-11T21:40:54.5188424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5188598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5188979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5189169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5189189Z 
2023-01-11T21:40:54.5189295Z Running tests...
2023-01-11T21:40:54.5189556Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5189867Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5190315Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5190337Z 
2023-01-11T21:40:54.5190597Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5190691Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5190713Z 
2023-01-11T21:40:54.5190820Z OK (skipped=1)
2023-01-11T21:40:54.5190840Z 
2023-01-11T21:40:54.5190962Z Generating XML reports...
2023-01-11T21:40:54.5191406Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212820.xml
2023-01-11T21:40:54.5191773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5191947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5192324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5192514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5192534Z 
2023-01-11T21:40:54.5192623Z Running tests...
2023-01-11T21:40:54.5192884Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5193193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5193641Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5193661Z 
2023-01-11T21:40:54.5193916Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5194025Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5194045Z 
2023-01-11T21:40:54.5194150Z OK (skipped=1)
2023-01-11T21:40:54.5194173Z 
2023-01-11T21:40:54.5194296Z Generating XML reports...
2023-01-11T21:40:54.5194743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212822.xml
2023-01-11T21:40:54.5195112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5195269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5195711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5195904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5195924Z 
2023-01-11T21:40:54.5196033Z Running tests...
2023-01-11T21:40:54.5196296Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5196610Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5197097Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5197119Z 
2023-01-11T21:40:54.5197385Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5197500Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5197520Z 
2023-01-11T21:40:54.5197610Z OK (skipped=1)
2023-01-11T21:40:54.5197629Z 
2023-01-11T21:40:54.5197753Z Generating XML reports...
2023-01-11T21:40:54.5198199Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212825.xml
2023-01-11T21:40:54.5198569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5198745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5199125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5199315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5199334Z 
2023-01-11T21:40:54.5199652Z Running tests...
2023-01-11T21:40:54.5199905Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5200226Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5200669Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5200690Z 
2023-01-11T21:40:54.5200948Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5201058Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5201077Z 
2023-01-11T21:40:54.5201188Z OK (skipped=1)
2023-01-11T21:40:54.5201207Z 
2023-01-11T21:40:54.5201329Z Generating XML reports...
2023-01-11T21:40:54.5201771Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212827.xml
2023-01-11T21:40:54.5202137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5202316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5202674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5202863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5202882Z 
2023-01-11T21:40:54.5202988Z Running tests...
2023-01-11T21:40:54.5203246Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5203557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5204004Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5204024Z 
2023-01-11T21:40:54.5204283Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5204475Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5204495Z 
2023-01-11T21:40:54.5204601Z OK (skipped=1)
2023-01-11T21:40:54.5204621Z 
2023-01-11T21:40:54.5204727Z Generating XML reports...
2023-01-11T21:40:54.5205178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212830.xml
2023-01-11T21:40:54.5205544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5205774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5206167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5206356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5206377Z 
2023-01-11T21:40:54.5206485Z Running tests...
2023-01-11T21:40:54.5206742Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5207060Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5207430Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5207467Z 
2023-01-11T21:40:54.5207709Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5207819Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5207838Z 
2023-01-11T21:40:54.5207946Z OK (skipped=1)
2023-01-11T21:40:54.5207965Z 
2023-01-11T21:40:54.5208089Z Generating XML reports...
2023-01-11T21:40:54.5208537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212832.xml
2023-01-11T21:40:54.5208906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5209085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5209465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5209638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5209657Z 
2023-01-11T21:40:54.5209766Z Running tests...
2023-01-11T21:40:54.5210023Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5210336Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5210722Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:40:54.5210743Z 
2023-01-11T21:40:54.5211003Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5211119Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5211138Z 
2023-01-11T21:40:54.5211244Z OK (skipped=1)
2023-01-11T21:40:54.5211263Z 
2023-01-11T21:40:54.5211384Z Generating XML reports...
2023-01-11T21:40:54.5211813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212834.xml
2023-01-11T21:40:54.5212183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5212357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5212737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5212928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5212947Z 
2023-01-11T21:40:54.5213095Z Running tests...
2023-01-11T21:40:54.5213361Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5213735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5213988Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5224840Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2023-01-11T21:40:54.5224884Z 
2023-01-11T21:40:54.5225242Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5225371Z Ran 1 test in 1.614s
2023-01-11T21:40:54.5225391Z 
2023-01-11T21:40:54.5225502Z OK (skipped=1)
2023-01-11T21:40:54.5225520Z 
2023-01-11T21:40:54.5225646Z Generating XML reports...
2023-01-11T21:40:54.5226111Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212837.xml
2023-01-11T21:40:54.5226493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5226669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5227053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5227226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5227264Z 
2023-01-11T21:40:54.5227360Z Running tests...
2023-01-11T21:40:54.5227623Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5227939Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5228192Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5228410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12652
2023-01-11T21:40:54.5228628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12653
2023-01-11T21:40:54.5229000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5229172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5229534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5229726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5230093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5230264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5230635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5230825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5231072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5231319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5231704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5232106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5232335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5232563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5232663Z ok (6.721s)
2023-01-11T21:40:54.5232684Z 
2023-01-11T21:40:54.5232947Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5233123Z Ran 1 test in 6.721s
2023-01-11T21:40:54.5233143Z 
2023-01-11T21:40:54.5233234Z OK
2023-01-11T21:40:54.5233254Z 
2023-01-11T21:40:54.5233378Z Generating XML reports...
2023-01-11T21:40:54.5233817Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212841.xml
2023-01-11T21:40:54.5234185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5234407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5234794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5234984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5235004Z 
2023-01-11T21:40:54.5235111Z Running tests...
2023-01-11T21:40:54.5235370Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5235685Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5235941Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5236159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12766
2023-01-11T21:40:54.5236372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12767
2023-01-11T21:40:54.5236744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5236916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5237292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5237480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5237843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5238014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5238374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5238560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5238809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5239051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5239689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5240103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5240339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5240563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5240799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5241017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5241421Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp4usxfv5t'>
2023-01-11T21:40:54.5241585Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:40:54.5241976Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp004iiggh'>
2023-01-11T21:40:54.5242139Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:40:54.5242240Z ok (7.440s)
2023-01-11T21:40:54.5242353Z 
2023-01-11T21:40:54.5242628Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5242741Z Ran 1 test in 7.440s
2023-01-11T21:40:54.5242760Z 
2023-01-11T21:40:54.5242834Z OK
2023-01-11T21:40:54.5242853Z 
2023-01-11T21:40:54.5242975Z Generating XML reports...
2023-01-11T21:40:54.5243426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212850.xml
2023-01-11T21:40:54.5243793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5244028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5244419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5244609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5244629Z 
2023-01-11T21:40:54.5244736Z Running tests...
2023-01-11T21:40:54.5245002Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5245297Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5245572Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.005s)
2023-01-11T21:40:54.5245592Z 
2023-01-11T21:40:54.5245849Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5245959Z Ran 1 test in 0.005s
2023-01-11T21:40:54.5245979Z 
2023-01-11T21:40:54.5246088Z OK (skipped=1)
2023-01-11T21:40:54.5246108Z 
2023-01-11T21:40:54.5246228Z Generating XML reports...
2023-01-11T21:40:54.5246675Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212900.xml
2023-01-11T21:40:54.5247047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5247208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5247589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5247779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5247799Z 
2023-01-11T21:40:54.5247905Z Running tests...
2023-01-11T21:40:54.5248162Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5248475Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5248740Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5248958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12918
2023-01-11T21:40:54.5249174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12919
2023-01-11T21:40:54.5249529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5249707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5250072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5250244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5250619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5250811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5251188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5251375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5251604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5251932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5252336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5252729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5252958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5253227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5253470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5253704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5253806Z ok (7.338s)
2023-01-11T21:40:54.5253826Z 
2023-01-11T21:40:54.5254078Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5254188Z Ran 1 test in 7.338s
2023-01-11T21:40:54.5254208Z 
2023-01-11T21:40:54.5254300Z OK
2023-01-11T21:40:54.5254319Z 
2023-01-11T21:40:54.5254441Z Generating XML reports...
2023-01-11T21:40:54.5254888Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212903.xml
2023-01-11T21:40:54.5255256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5255434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5255815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5256004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5256024Z 
2023-01-11T21:40:54.5256115Z Running tests...
2023-01-11T21:40:54.5256377Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5256690Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5256974Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5257192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13037
2023-01-11T21:40:54.5257409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13038
2023-01-11T21:40:54.5257781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5257955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5258319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5258511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5258877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5259046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5259418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5259605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5259851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5260093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5260490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5260867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5261154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5261379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5261618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5261859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5262320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5262721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5262957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.5263194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.5263564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5263954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5264056Z ok (5.825s)
2023-01-11T21:40:54.5264076Z 
2023-01-11T21:40:54.5264337Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5264450Z Ran 1 test in 5.825s
2023-01-11T21:40:54.5264469Z 
2023-01-11T21:40:54.5264559Z OK
2023-01-11T21:40:54.5264579Z 
2023-01-11T21:40:54.5264702Z Generating XML reports...
2023-01-11T21:40:54.5265150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212913.xml
2023-01-11T21:40:54.5265520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5265683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5266059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5266248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5266268Z 
2023-01-11T21:40:54.5266375Z Running tests...
2023-01-11T21:40:54.5266634Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5266947Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5267224Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5267440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13159
2023-01-11T21:40:54.5267641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13160
2023-01-11T21:40:54.5268014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5268191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5268566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5268755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5269117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5269287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5269667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5269851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5270134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5270376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5270778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5271169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5271474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5271708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5271949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5272187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5272583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5272960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5273199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.5273437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.5273830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5274217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5274318Z ok (16.661s)
2023-01-11T21:40:54.5274338Z 
2023-01-11T21:40:54.5274598Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5274713Z Ran 1 test in 16.662s
2023-01-11T21:40:54.5274733Z 
2023-01-11T21:40:54.5274823Z OK
2023-01-11T21:40:54.5274842Z 
2023-01-11T21:40:54.5274949Z Generating XML reports...
2023-01-11T21:40:54.5275398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212921.xml
2023-01-11T21:40:54.5275771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5275947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5276325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5276514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5276533Z 
2023-01-11T21:40:54.5276641Z Running tests...
2023-01-11T21:40:54.5276899Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5277198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5277508Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5277724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13281
2023-01-11T21:40:54.5277939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13282
2023-01-11T21:40:54.5278312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5278485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5278850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5279021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5279775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5279962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5280346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5280538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5280865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5281120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5281521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5281915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5282149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5282378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5282463Z ok (7.741s)
2023-01-11T21:40:54.5282483Z 
2023-01-11T21:40:54.5282744Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5282854Z Ran 1 test in 7.741s
2023-01-11T21:40:54.5282874Z 
2023-01-11T21:40:54.5282965Z OK
2023-01-11T21:40:54.5282985Z 
2023-01-11T21:40:54.5283110Z Generating XML reports...
2023-01-11T21:40:54.5283562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212940.xml
2023-01-11T21:40:54.5283931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5284108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5284476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5284666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5284686Z 
2023-01-11T21:40:54.5284795Z Running tests...
2023-01-11T21:40:54.5285059Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5285373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5285668Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5285885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13400
2023-01-11T21:40:54.5286102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13401
2023-01-11T21:40:54.5286455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5286635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5287016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5287203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5287568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5287739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5288110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5288295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5288542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5288838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5289243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5289634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5289862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5290131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5290238Z ok (7.761s)
2023-01-11T21:40:54.5290258Z 
2023-01-11T21:40:54.5290522Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5290633Z Ran 1 test in 7.761s
2023-01-11T21:40:54.5290653Z 
2023-01-11T21:40:54.5290744Z OK
2023-01-11T21:40:54.5290764Z 
2023-01-11T21:40:54.5290869Z Generating XML reports...
2023-01-11T21:40:54.5291320Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212950.xml
2023-01-11T21:40:54.5291688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5291859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5292235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5292427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5292447Z 
2023-01-11T21:40:54.5292553Z Running tests...
2023-01-11T21:40:54.5292816Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5293113Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5293371Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5293590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13519
2023-01-11T21:40:54.5293804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13520
2023-01-11T21:40:54.5294173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5294345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5294724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5294914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5295278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5295433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5295808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5295992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5296234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5296633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5296874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5297270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5297496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5297722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5297863Z ok (7.487s)
2023-01-11T21:40:54.5297883Z 
2023-01-11T21:40:54.5298146Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5298258Z Ran 1 test in 7.487s
2023-01-11T21:40:54.5298278Z 
2023-01-11T21:40:54.5298368Z OK
2023-01-11T21:40:54.5298387Z 
2023-01-11T21:40:54.5298511Z Generating XML reports...
2023-01-11T21:40:54.5298959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213001.xml
2023-01-11T21:40:54.5299372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5299553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5299917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5300105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5300129Z 
2023-01-11T21:40:54.5300235Z Running tests...
2023-01-11T21:40:54.5300493Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5300802Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5301064Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5301280Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13634
2023-01-11T21:40:54.5301499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13635
2023-01-11T21:40:54.5301855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5302028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5302406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5302598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5302959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5303130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5303496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5303682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5303929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5304154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5304554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5304953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5305178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5305406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5306189Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:40:54.5306972Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:40:54.5307264Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5307538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5307643Z ok (7.463s)
2023-01-11T21:40:54.5307664Z 
2023-01-11T21:40:54.5307931Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5308042Z Ran 1 test in 7.463s
2023-01-11T21:40:54.5308062Z 
2023-01-11T21:40:54.5308136Z OK
2023-01-11T21:40:54.5308172Z 
2023-01-11T21:40:54.5308283Z Generating XML reports...
2023-01-11T21:40:54.5308731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213011.xml
2023-01-11T21:40:54.5309100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5309274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5309653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5309844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5309864Z 
2023-01-11T21:40:54.5309970Z Running tests...
2023-01-11T21:40:54.5310228Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5310525Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5310805Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5311562Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.652s)
2023-01-11T21:40:54.5311583Z 
2023-01-11T21:40:54.5311839Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5311952Z Ran 1 test in 1.652s
2023-01-11T21:40:54.5311972Z 
2023-01-11T21:40:54.5312078Z OK (skipped=1)
2023-01-11T21:40:54.5312096Z 
2023-01-11T21:40:54.5312219Z Generating XML reports...
2023-01-11T21:40:54.5312667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213021.xml
2023-01-11T21:40:54.5313030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5313256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5313624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5313814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5313833Z 
2023-01-11T21:40:54.5313939Z Running tests...
2023-01-11T21:40:54.5314197Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5314510Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5314790Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5315533Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2023-01-11T21:40:54.5315608Z 
2023-01-11T21:40:54.5315876Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5315987Z Ran 1 test in 1.643s
2023-01-11T21:40:54.5316006Z 
2023-01-11T21:40:54.5316095Z OK (skipped=1)
2023-01-11T21:40:54.5316131Z 
2023-01-11T21:40:54.5316238Z Generating XML reports...
2023-01-11T21:40:54.5316746Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213025.xml
2023-01-11T21:40:54.5317127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5317302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5317678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5317872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5317892Z 
2023-01-11T21:40:54.5317998Z Running tests...
2023-01-11T21:40:54.5318259Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5318552Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5318829Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5319051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13821
2023-01-11T21:40:54.5319269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13822
2023-01-11T21:40:54.5319967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5320144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5320530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5320717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5321063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5321236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5321610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5321796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5322039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5322281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5322681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5323081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5323310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5323519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5323858Z STAGE:2023-01-11 21:30:35 13821:13821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5324179Z STAGE:2023-01-11 21:30:35 13822:13822 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5324414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5324646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5325070Z STAGE:2023-01-11 21:30:35 13821:13821 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5325402Z STAGE:2023-01-11 21:30:35 13822:13822 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5325749Z STAGE:2023-01-11 21:30:35 13821:13821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5326094Z STAGE:2023-01-11 21:30:35 13822:13822 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5326948Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:40:54.5327742Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:40:54.5328081Z STAGE:2023-01-11 21:30:36 13821:13821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5328401Z STAGE:2023-01-11 21:30:36 13822:13822 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5328715Z STAGE:2023-01-11 21:30:36 13821:13821 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5329045Z STAGE:2023-01-11 21:30:36 13822:13822 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5329384Z STAGE:2023-01-11 21:30:36 13821:13821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5329727Z STAGE:2023-01-11 21:30:36 13822:13822 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5329827Z ok (6.757s)
2023-01-11T21:40:54.5329848Z 
2023-01-11T21:40:54.5330109Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5330219Z Ran 1 test in 6.758s
2023-01-11T21:40:54.5330242Z 
2023-01-11T21:40:54.5330333Z OK
2023-01-11T21:40:54.5330353Z 
2023-01-11T21:40:54.5330458Z Generating XML reports...
2023-01-11T21:40:54.5330908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213029.xml
2023-01-11T21:40:54.5331277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5331456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5331836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5332026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5332046Z 
2023-01-11T21:40:54.5332152Z Running tests...
2023-01-11T21:40:54.5332413Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5332731Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5332983Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5333202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13943
2023-01-11T21:40:54.5333420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13944
2023-01-11T21:40:54.5333851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5334023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5334399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5334585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5334991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5335151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5335528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5335713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5335955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5336203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5336603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5336996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5337230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5337455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5337539Z ok (6.845s)
2023-01-11T21:40:54.5337559Z 
2023-01-11T21:40:54.5337821Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5337933Z Ran 1 test in 6.845s
2023-01-11T21:40:54.5337953Z 
2023-01-11T21:40:54.5338047Z OK
2023-01-11T21:40:54.5338067Z 
2023-01-11T21:40:54.5338189Z Generating XML reports...
2023-01-11T21:40:54.5338638Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213039.xml
2023-01-11T21:40:54.5339005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5339178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5339559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5339732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5339752Z 
2023-01-11T21:40:54.5339858Z Running tests...
2023-01-11T21:40:54.5340116Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5340425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5340702Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5341445Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.639s)
2023-01-11T21:40:54.5341466Z 
2023-01-11T21:40:54.5341725Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5341835Z Ran 1 test in 1.640s
2023-01-11T21:40:54.5341855Z 
2023-01-11T21:40:54.5341961Z OK (skipped=1)
2023-01-11T21:40:54.5341980Z 
2023-01-11T21:40:54.5342085Z Generating XML reports...
2023-01-11T21:40:54.5342531Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213048.xml
2023-01-11T21:40:54.5342897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5343127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5343506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5343694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5343714Z 
2023-01-11T21:40:54.5343820Z Running tests...
2023-01-11T21:40:54.5344121Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5344440Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5344704Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5344920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14092
2023-01-11T21:40:54.5345140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14093
2023-01-11T21:40:54.5345509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5345681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5346058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5346248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5346614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5346766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5347141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5347328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5347576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5347816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5348217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5348611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5348842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5349070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5349989Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T21:40:54.5350090Z   warnings.warn(
2023-01-11T21:40:54.5351000Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T21:40:54.5351114Z   warnings.warn(
2023-01-11T21:40:54.5351349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5351578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5351678Z ok (6.342s)
2023-01-11T21:40:54.5351748Z 
2023-01-11T21:40:54.5352015Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5352126Z Ran 1 test in 6.342s
2023-01-11T21:40:54.5352146Z 
2023-01-11T21:40:54.5352236Z OK
2023-01-11T21:40:54.5352255Z 
2023-01-11T21:40:54.5352362Z Generating XML reports...
2023-01-11T21:40:54.5352812Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213052.xml
2023-01-11T21:40:54.5353229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5353411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5353790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5353984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5354004Z 
2023-01-11T21:40:54.5354114Z Running tests...
2023-01-11T21:40:54.5354372Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5354685Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5354945Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5355691Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2023-01-11T21:40:54.5355712Z 
2023-01-11T21:40:54.5355968Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5356078Z Ran 1 test in 1.629s
2023-01-11T21:40:54.5356097Z 
2023-01-11T21:40:54.5356205Z OK (skipped=1)
2023-01-11T21:40:54.5356224Z 
2023-01-11T21:40:54.5356350Z Generating XML reports...
2023-01-11T21:40:54.5356795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213101.xml
2023-01-11T21:40:54.5357162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5357336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5357716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5357894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5357914Z 
2023-01-11T21:40:54.5358024Z Running tests...
2023-01-11T21:40:54.5358282Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5358593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5358864Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5359085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14244
2023-01-11T21:40:54.5359299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14245
2023-01-11T21:40:54.5359920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5360080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5360466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5360654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5361019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5361191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5361650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5361839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5362083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5362333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5362770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5363179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5363410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5363638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5363977Z STAGE:2023-01-11 21:31:11 14245:14245 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5364299Z STAGE:2023-01-11 21:31:11 14244:14244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5364534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5364763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:40:54.5365083Z STAGE:2023-01-11 21:31:11 14245:14245 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5365410Z STAGE:2023-01-11 21:31:11 14244:14244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5365754Z STAGE:2023-01-11 21:31:11 14245:14245 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5366099Z STAGE:2023-01-11 21:31:11 14244:14244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5366426Z STAGE:2023-01-11 21:31:11 14244:14244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5366756Z STAGE:2023-01-11 21:31:12 14244:14244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5367101Z STAGE:2023-01-11 21:31:12 14244:14244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5367203Z ok (7.163s)
2023-01-11T21:40:54.5367223Z 
2023-01-11T21:40:54.5367486Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5367585Z Ran 1 test in 7.163s
2023-01-11T21:40:54.5367605Z 
2023-01-11T21:40:54.5367697Z OK
2023-01-11T21:40:54.5367717Z 
2023-01-11T21:40:54.5367839Z Generating XML reports...
2023-01-11T21:40:54.5368289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213105.xml
2023-01-11T21:40:54.5368660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5368840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5369218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5369407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5369427Z 
2023-01-11T21:40:54.5369535Z Running tests...
2023-01-11T21:40:54.5369782Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5370099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5370364Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5370582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14366
2023-01-11T21:40:54.5370800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14367
2023-01-11T21:40:54.5371227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5371401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5371781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5371951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5372357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5372533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5372906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5373093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5373342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5373585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5373983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5374376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5374593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5374820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5374918Z ok (6.971s)
2023-01-11T21:40:54.5374938Z 
2023-01-11T21:40:54.5375202Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5375313Z Ran 1 test in 6.971s
2023-01-11T21:40:54.5375335Z 
2023-01-11T21:40:54.5375426Z OK
2023-01-11T21:40:54.5375445Z 
2023-01-11T21:40:54.5375567Z Generating XML reports...
2023-01-11T21:40:54.5376017Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213115.xml
2023-01-11T21:40:54.5376367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5376542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5376919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5377109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5377129Z 
2023-01-11T21:40:54.5377235Z Running tests...
2023-01-11T21:40:54.5377494Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5377804Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5378081Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5378298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14481
2023-01-11T21:40:54.5378498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14482
2023-01-11T21:40:54.5378866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5379043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5379419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5379606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5379966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5380196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5380573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5380744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5380990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5381276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5381683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5382073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5382303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5382532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5382633Z ok (6.843s)
2023-01-11T21:40:54.5382653Z 
2023-01-11T21:40:54.5382917Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5383011Z Ran 1 test in 6.843s
2023-01-11T21:40:54.5383030Z 
2023-01-11T21:40:54.5383123Z OK
2023-01-11T21:40:54.5383142Z 
2023-01-11T21:40:54.5383264Z Generating XML reports...
2023-01-11T21:40:54.5383717Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213125.xml
2023-01-11T21:40:54.5384084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5384258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5384633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5384827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5384847Z 
2023-01-11T21:40:54.5384955Z Running tests...
2023-01-11T21:40:54.5385200Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5385511Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5385786Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5386537Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.610s)
2023-01-11T21:40:54.5386558Z 
2023-01-11T21:40:54.5386818Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5386933Z Ran 1 test in 1.610s
2023-01-11T21:40:54.5386953Z 
2023-01-11T21:40:54.5387058Z OK (skipped=1)
2023-01-11T21:40:54.5387077Z 
2023-01-11T21:40:54.5387199Z Generating XML reports...
2023-01-11T21:40:54.5387643Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213134.xml
2023-01-11T21:40:54.5387994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5388172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5388552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5388741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5388761Z 
2023-01-11T21:40:54.5388866Z Running tests...
2023-01-11T21:40:54.5389127Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5389502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5389759Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5390559Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2023-01-11T21:40:54.5390583Z 
2023-01-11T21:40:54.5390848Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5390941Z Ran 1 test in 1.634s
2023-01-11T21:40:54.5390961Z 
2023-01-11T21:40:54.5391067Z OK (skipped=1)
2023-01-11T21:40:54.5391086Z 
2023-01-11T21:40:54.5391206Z Generating XML reports...
2023-01-11T21:40:54.5391653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213138.xml
2023-01-11T21:40:54.5392030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5392205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5392581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5392769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5392792Z 
2023-01-11T21:40:54.5392899Z Running tests...
2023-01-11T21:40:54.5393143Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5393452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5393740Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5394486Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2023-01-11T21:40:54.5394506Z 
2023-01-11T21:40:54.5394766Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5394876Z Ran 1 test in 1.628s
2023-01-11T21:40:54.5394895Z 
2023-01-11T21:40:54.5395000Z OK (skipped=1)
2023-01-11T21:40:54.5395022Z 
2023-01-11T21:40:54.5395144Z Generating XML reports...
2023-01-11T21:40:54.5395590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213142.xml
2023-01-11T21:40:54.5395941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5396115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5396496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5396684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5396704Z 
2023-01-11T21:40:54.5396807Z Running tests...
2023-01-11T21:40:54.5397065Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5397375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5397674Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5397892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14698
2023-01-11T21:40:54.5398093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14699
2023-01-11T21:40:54.5398459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5398689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5399068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5399257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5399844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5400089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5400479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5400650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5400896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5401144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5401542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5401936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5402164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5402395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5402496Z ok (7.351s)
2023-01-11T21:40:54.5402516Z 
2023-01-11T21:40:54.5402776Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5402869Z Ran 1 test in 7.351s
2023-01-11T21:40:54.5402889Z 
2023-01-11T21:40:54.5402980Z OK
2023-01-11T21:40:54.5403003Z 
2023-01-11T21:40:54.5403124Z Generating XML reports...
2023-01-11T21:40:54.5403571Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213146.xml
2023-01-11T21:40:54.5403938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5404112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5404487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5404679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5404699Z 
2023-01-11T21:40:54.5404806Z Running tests...
2023-01-11T21:40:54.5405047Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5405357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5405629Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5405845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14817
2023-01-11T21:40:54.5406062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14818
2023-01-11T21:40:54.5406432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5406603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5406983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5407155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5407520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5407691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5408140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5408327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5408571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5408813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5409256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5409661Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5409873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5410250Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T21:40:54.5410508Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T21:40:54.5410736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5411102Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T21:40:54.5411347Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T21:40:54.5411447Z ok (7.018s)
2023-01-11T21:40:54.5411471Z 
2023-01-11T21:40:54.5411730Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5411824Z Ran 1 test in 7.018s
2023-01-11T21:40:54.5411860Z 
2023-01-11T21:40:54.5411936Z OK
2023-01-11T21:40:54.5411955Z 
2023-01-11T21:40:54.5412076Z Generating XML reports...
2023-01-11T21:40:54.5412524Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213156.xml
2023-01-11T21:40:54.5412896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5413069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5413493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5413684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5413704Z 
2023-01-11T21:40:54.5413815Z Running tests...
2023-01-11T21:40:54.5414057Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5414369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5414629Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5414846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14932
2023-01-11T21:40:54.5415064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14933
2023-01-11T21:40:54.5415434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5415605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5415983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5416161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5416526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5416697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5417069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5417347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5417593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5417834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5418236Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5418669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5418883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5419130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5419353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5419592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5419989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5420376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5420476Z ok (4.324s)
2023-01-11T21:40:54.5420496Z 
2023-01-11T21:40:54.5420762Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5420873Z Ran 1 test in 4.324s
2023-01-11T21:40:54.5420893Z 
2023-01-11T21:40:54.5420967Z OK
2023-01-11T21:40:54.5420986Z 
2023-01-11T21:40:54.5421110Z Generating XML reports...
2023-01-11T21:40:54.5421558Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213206.xml
2023-01-11T21:40:54.5421930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5422109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5422488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5422678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5422698Z 
2023-01-11T21:40:54.5422804Z Running tests...
2023-01-11T21:40:54.5423051Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5423364Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5423615Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5423831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15039
2023-01-11T21:40:54.5424047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15040
2023-01-11T21:40:54.5424417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5424590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5424966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5425154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5425503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5425676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5426046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5426231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5426530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5426775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5427175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5427568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5427839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5428070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5428291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5428523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5428923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5429312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5429413Z ok (4.434s)
2023-01-11T21:40:54.5429433Z 
2023-01-11T21:40:54.5429693Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5429804Z Ran 1 test in 4.434s
2023-01-11T21:40:54.5429824Z 
2023-01-11T21:40:54.5429901Z OK
2023-01-11T21:40:54.5429938Z 
2023-01-11T21:40:54.5430044Z Generating XML reports...
2023-01-11T21:40:54.5430491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213213.xml
2023-01-11T21:40:54.5430856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5431034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5431410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5431601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5431620Z 
2023-01-11T21:40:54.5431726Z Running tests...
2023-01-11T21:40:54.5431986Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5432286Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5432561Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5433309Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T21:40:54.5433333Z 
2023-01-11T21:40:54.5433588Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5433699Z Ran 1 test in 1.633s
2023-01-11T21:40:54.5433718Z 
2023-01-11T21:40:54.5433825Z OK (skipped=1)
2023-01-11T21:40:54.5433844Z 
2023-01-11T21:40:54.5433965Z Generating XML reports...
2023-01-11T21:40:54.5434407Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213220.xml
2023-01-11T21:40:54.5434781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5434953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5435314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5435505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5435576Z 
2023-01-11T21:40:54.5435687Z Running tests...
2023-01-11T21:40:54.5435949Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5436261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5436537Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5437324Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2023-01-11T21:40:54.5437347Z 
2023-01-11T21:40:54.5437609Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5437719Z Ran 1 test in 1.626s
2023-01-11T21:40:54.5437743Z 
2023-01-11T21:40:54.5437832Z OK (skipped=1)
2023-01-11T21:40:54.5437869Z 
2023-01-11T21:40:54.5437976Z Generating XML reports...
2023-01-11T21:40:54.5438419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213224.xml
2023-01-11T21:40:54.5438787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5438959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5439339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5439746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5439768Z 
2023-01-11T21:40:54.5439880Z Running tests...
2023-01-11T21:40:54.5440147Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5440444Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5440723Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5440938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15214
2023-01-11T21:40:54.5441154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15215
2023-01-11T21:40:54.5441520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5441698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5442076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5442264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5442612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5442788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5443153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5443338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5443583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5443829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5444226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5444618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5444847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5445136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5445239Z ok (4.329s)
2023-01-11T21:40:54.5445259Z 
2023-01-11T21:40:54.5445522Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5445631Z Ran 1 test in 4.329s
2023-01-11T21:40:54.5445651Z 
2023-01-11T21:40:54.5445742Z OK
2023-01-11T21:40:54.5445761Z 
2023-01-11T21:40:54.5445885Z Generating XML reports...
2023-01-11T21:40:54.5446390Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213228.xml
2023-01-11T21:40:54.5446773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5446948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5447308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5447501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5447521Z 
2023-01-11T21:40:54.5447626Z Running tests...
2023-01-11T21:40:54.5447886Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5448199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5448449Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5448470Z 
2023-01-11T21:40:54.5448727Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5448838Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5448858Z 
2023-01-11T21:40:54.5448947Z OK (skipped=1)
2023-01-11T21:40:54.5448983Z 
2023-01-11T21:40:54.5449088Z Generating XML reports...
2023-01-11T21:40:54.5449535Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213235.xml
2023-01-11T21:40:54.5449909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5450081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5450459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5450649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5450668Z 
2023-01-11T21:40:54.5450774Z Running tests...
2023-01-11T21:40:54.5451038Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5451334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5451593Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5451613Z 
2023-01-11T21:40:54.5451873Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5451981Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5452002Z 
2023-01-11T21:40:54.5452107Z OK (skipped=1)
2023-01-11T21:40:54.5452126Z 
2023-01-11T21:40:54.5452247Z Generating XML reports...
2023-01-11T21:40:54.5452691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213237.xml
2023-01-11T21:40:54.5453061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5453238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5453601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5453790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5453809Z 
2023-01-11T21:40:54.5453915Z Running tests...
2023-01-11T21:40:54.5454228Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5454538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5454787Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5455004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15385
2023-01-11T21:40:54.5455218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15386
2023-01-11T21:40:54.5455630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5455812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5456190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5456378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5456742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5456913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5457281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5457468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5457715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5457941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5458338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5458733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5458964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5459192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5459527Z STAGE:2023-01-11 21:32:45 15385:15385 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5459849Z STAGE:2023-01-11 21:32:45 15386:15386 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5460182Z STAGE:2023-01-11 21:32:47 15385:15385 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5460530Z STAGE:2023-01-11 21:32:47 15385:15385 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5460846Z STAGE:2023-01-11 21:32:47 15386:15386 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5461190Z STAGE:2023-01-11 21:32:47 15386:15386 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5461521Z STAGE:2023-01-11 21:32:47 15385:15385 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5461839Z STAGE:2023-01-11 21:32:47 15386:15386 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5462167Z STAGE:2023-01-11 21:32:47 15386:15386 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5462510Z STAGE:2023-01-11 21:32:47 15386:15386 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5462839Z STAGE:2023-01-11 21:32:47 15385:15385 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5463179Z STAGE:2023-01-11 21:32:47 15385:15385 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5463265Z ok (7.757s)
2023-01-11T21:40:54.5463300Z 
2023-01-11T21:40:54.5463549Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5463714Z Ran 1 test in 7.757s
2023-01-11T21:40:54.5463734Z 
2023-01-11T21:40:54.5463826Z OK
2023-01-11T21:40:54.5463846Z 
2023-01-11T21:40:54.5463969Z Generating XML reports...
2023-01-11T21:40:54.5464421Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213240.xml
2023-01-11T21:40:54.5464788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5464963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5465382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5465563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5465600Z 
2023-01-11T21:40:54.5465691Z Running tests...
2023-01-11T21:40:54.5465953Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5466271Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5466534Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5466555Z 
2023-01-11T21:40:54.5466808Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5466919Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5466938Z 
2023-01-11T21:40:54.5467047Z OK (skipped=1)
2023-01-11T21:40:54.5467066Z 
2023-01-11T21:40:54.5467186Z Generating XML reports...
2023-01-11T21:40:54.5467621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213250.xml
2023-01-11T21:40:54.5467994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5468167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5468545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5468740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5468760Z 
2023-01-11T21:40:54.5468866Z Running tests...
2023-01-11T21:40:54.5469126Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5469438Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5469682Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5469721Z 
2023-01-11T21:40:54.5469964Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5470074Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5470093Z 
2023-01-11T21:40:54.5470198Z OK (skipped=1)
2023-01-11T21:40:54.5470216Z 
2023-01-11T21:40:54.5470337Z Generating XML reports...
2023-01-11T21:40:54.5470781Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213253.xml
2023-01-11T21:40:54.5471153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5471326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5471704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5471877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5471917Z 
2023-01-11T21:40:54.5472008Z Running tests...
2023-01-11T21:40:54.5472267Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5472577Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5472829Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5473101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15574
2023-01-11T21:40:54.5473316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15575
2023-01-11T21:40:54.5473746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5473920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5474284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5474521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5474891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5475061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5475429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5475617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5475863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5476106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5476489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5476890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5477117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5477344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5477445Z ok (7.061s)
2023-01-11T21:40:54.5477467Z 
2023-01-11T21:40:54.5477729Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5477838Z Ran 1 test in 7.061s
2023-01-11T21:40:54.5477858Z 
2023-01-11T21:40:54.5477950Z OK
2023-01-11T21:40:54.5477969Z 
2023-01-11T21:40:54.5478091Z Generating XML reports...
2023-01-11T21:40:54.5478522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213255.xml
2023-01-11T21:40:54.5478891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5479066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5479673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5479871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5479893Z 
2023-01-11T21:40:54.5480002Z Running tests...
2023-01-11T21:40:54.5480270Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5480583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5480834Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5481587Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2023-01-11T21:40:54.5481625Z 
2023-01-11T21:40:54.5481869Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5481977Z Ran 1 test in 1.642s
2023-01-11T21:40:54.5481996Z 
2023-01-11T21:40:54.5482102Z OK (skipped=1)
2023-01-11T21:40:54.5482121Z 
2023-01-11T21:40:54.5482244Z Generating XML reports...
2023-01-11T21:40:54.5482779Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213305.xml
2023-01-11T21:40:54.5483149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5483321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5483698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5483926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5483966Z 
2023-01-11T21:40:54.5484062Z Running tests...
2023-01-11T21:40:54.5484324Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5484637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5484884Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5485105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15726
2023-01-11T21:40:54.5485321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15727
2023-01-11T21:40:54.5485692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5485849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5486230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5486417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5486780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5486953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5487327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5487515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5487759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5488005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5488392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5488792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5489022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5489261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5489486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5489722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5490116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5490505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5490608Z ok (4.347s)
2023-01-11T21:40:54.5490631Z 
2023-01-11T21:40:54.5490878Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5490990Z Ran 1 test in 4.347s
2023-01-11T21:40:54.5491009Z 
2023-01-11T21:40:54.5491101Z OK
2023-01-11T21:40:54.5491120Z 
2023-01-11T21:40:54.5491243Z Generating XML reports...
2023-01-11T21:40:54.5491692Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213309.xml
2023-01-11T21:40:54.5492119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5492294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5492669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5492859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5492878Z 
2023-01-11T21:40:54.5493014Z Running tests...
2023-01-11T21:40:54.5493284Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5493596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5493839Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5494056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15833
2023-01-11T21:40:54.5494276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15834
2023-01-11T21:40:54.5494645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5494818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5495177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5495367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5495732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5495904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5496271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5496460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5496702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5496942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5497340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5497723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5497952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5498179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5498278Z ok (6.831s)
2023-01-11T21:40:54.5498298Z 
2023-01-11T21:40:54.5498569Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5498679Z Ran 1 test in 6.831s
2023-01-11T21:40:54.5498698Z 
2023-01-11T21:40:54.5498789Z OK
2023-01-11T21:40:54.5498808Z 
2023-01-11T21:40:54.5498931Z Generating XML reports...
2023-01-11T21:40:54.5499361Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213316.xml
2023-01-11T21:40:54.5499728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5499903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5500286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5500476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5500496Z 
2023-01-11T21:40:54.5500603Z Running tests...
2023-01-11T21:40:54.5500865Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5501234Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5501478Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5501679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15948
2023-01-11T21:40:54.5501892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15949
2023-01-11T21:40:54.5502303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5502482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5502861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5503047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5503409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5503581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5503933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5504119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5504363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5504609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5505003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5505395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5505627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5505854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5505954Z ok (4.536s)
2023-01-11T21:40:54.5505974Z 
2023-01-11T21:40:54.5506221Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5506331Z Ran 1 test in 4.537s
2023-01-11T21:40:54.5506351Z 
2023-01-11T21:40:54.5506441Z OK
2023-01-11T21:40:54.5506460Z 
2023-01-11T21:40:54.5506586Z Generating XML reports...
2023-01-11T21:40:54.5507038Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213325.xml
2023-01-11T21:40:54.5507406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5507581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5507963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5508135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5508172Z 
2023-01-11T21:40:54.5508263Z Running tests...
2023-01-11T21:40:54.5508520Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5508829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5509099Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5509318Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16053
2023-01-11T21:40:54.5509534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16054
2023-01-11T21:40:54.5509901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5510130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5510494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5510682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5511042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5511212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5511626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5511821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5512065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5512308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5512692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5513087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5513358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5513602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5513828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5514062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5514453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5514852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5514953Z ok (4.254s)
2023-01-11T21:40:54.5514974Z 
2023-01-11T21:40:54.5515235Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5515330Z Ran 1 test in 4.254s
2023-01-11T21:40:54.5515349Z 
2023-01-11T21:40:54.5515440Z OK
2023-01-11T21:40:54.5515459Z 
2023-01-11T21:40:54.5515580Z Generating XML reports...
2023-01-11T21:40:54.5516032Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213332.xml
2023-01-11T21:40:54.5516401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5516574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5516953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5517145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5517164Z 
2023-01-11T21:40:54.5517255Z Running tests...
2023-01-11T21:40:54.5517513Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5517823Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5518080Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5518300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16160
2023-01-11T21:40:54.5518520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16161
2023-01-11T21:40:54.5518889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5519062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5519696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5519897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5520263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5520434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5520890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5521085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5521330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5521570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5521975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5522350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5522578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5522818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5523044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5523279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5523672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5524062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5524165Z ok (4.349s)
2023-01-11T21:40:54.5524185Z 
2023-01-11T21:40:54.5524446Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5524540Z Ran 1 test in 4.349s
2023-01-11T21:40:54.5524559Z 
2023-01-11T21:40:54.5524651Z OK
2023-01-11T21:40:54.5524670Z 
2023-01-11T21:40:54.5524790Z Generating XML reports...
2023-01-11T21:40:54.5525238Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213339.xml
2023-01-11T21:40:54.5525608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5525781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5526157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5526347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5526367Z 
2023-01-11T21:40:54.5526474Z Running tests...
2023-01-11T21:40:54.5526718Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5527032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5527293Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5527513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16267
2023-01-11T21:40:54.5527730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16268
2023-01-11T21:40:54.5528098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5528270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5528646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5528889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5529258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5529430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5529803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5530031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5530281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5530522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5530919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5531315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5531528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5531756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5531854Z ok (6.448s)
2023-01-11T21:40:54.5531873Z 
2023-01-11T21:40:54.5532139Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5532248Z Ran 1 test in 6.448s
2023-01-11T21:40:54.5532267Z 
2023-01-11T21:40:54.5532358Z OK
2023-01-11T21:40:54.5532377Z 
2023-01-11T21:40:54.5532499Z Generating XML reports...
2023-01-11T21:40:54.5532946Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213346.xml
2023-01-11T21:40:54.5533297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5533476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5533852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5534042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5534062Z 
2023-01-11T21:40:54.5534168Z Running tests...
2023-01-11T21:40:54.5534431Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5534741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5534976Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2023-01-11T21:40:54.5534996Z 
2023-01-11T21:40:54.5535250Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5535344Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5535366Z 
2023-01-11T21:40:54.5535473Z OK (skipped=1)
2023-01-11T21:40:54.5535492Z 
2023-01-11T21:40:54.5535614Z Generating XML reports...
2023-01-11T21:40:54.5536060Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213355.xml
2023-01-11T21:40:54.5536428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5536600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5536980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5537166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5537186Z 
2023-01-11T21:40:54.5537292Z Running tests...
2023-01-11T21:40:54.5537536Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5537905Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5538140Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T21:40:54.5538159Z 
2023-01-11T21:40:54.5538414Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5538524Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5538543Z 
2023-01-11T21:40:54.5538649Z OK (skipped=1)
2023-01-11T21:40:54.5538668Z 
2023-01-11T21:40:54.5538789Z Generating XML reports...
2023-01-11T21:40:54.5539283Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213357.xml
2023-01-11T21:40:54.5539643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5539818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5540193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5540386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5540406Z 
2023-01-11T21:40:54.5540512Z Running tests...
2023-01-11T21:40:54.5540772Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5541084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5541351Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T21:40:54.5541371Z 
2023-01-11T21:40:54.5541627Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5541721Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5541740Z 
2023-01-11T21:40:54.5541846Z OK (skipped=1)
2023-01-11T21:40:54.5541865Z 
2023-01-11T21:40:54.5541985Z Generating XML reports...
2023-01-11T21:40:54.5542431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213400.xml
2023-01-11T21:40:54.5542799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5542973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5543347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5543536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5543559Z 
2023-01-11T21:40:54.5543663Z Running tests...
2023-01-11T21:40:54.5543904Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5544213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5544472Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T21:40:54.5544495Z 
2023-01-11T21:40:54.5544753Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5544863Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5544882Z 
2023-01-11T21:40:54.5544987Z OK (skipped=1)
2023-01-11T21:40:54.5545006Z 
2023-01-11T21:40:54.5545127Z Generating XML reports...
2023-01-11T21:40:54.5545569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213402.xml
2023-01-11T21:40:54.5545921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5546096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5546472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5546658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5546678Z 
2023-01-11T21:40:54.5546838Z Running tests...
2023-01-11T21:40:54.5547100Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5547408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5547688Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5547906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16517
2023-01-11T21:40:54.5548149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16518
2023-01-11T21:40:54.5548527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5548700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5549078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5549271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5549637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5549809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5550183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5550353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5550603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5550846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5551248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5551644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5551877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5552116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5552338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5552575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5552954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5553347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5553585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.5553825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.5554216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5554606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5554836Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:40:54.5554939Z ok (22.377s)
2023-01-11T21:40:54.5554962Z 
2023-01-11T21:40:54.5555227Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5555323Z Ran 1 test in 22.378s
2023-01-11T21:40:54.5555360Z 
2023-01-11T21:40:54.5555435Z OK
2023-01-11T21:40:54.5555454Z 
2023-01-11T21:40:54.5555576Z Generating XML reports...
2023-01-11T21:40:54.5556024Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213404.xml
2023-01-11T21:40:54.5556453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5556629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5557005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5557191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5557211Z 
2023-01-11T21:40:54.5557403Z Running tests...
2023-01-11T21:40:54.5557657Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5557968Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5558267Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5558489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16640
2023-01-11T21:40:54.5558704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16641
2023-01-11T21:40:54.5559072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5559245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5559850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5560047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5560404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5560573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5560942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5561133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5561377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5561620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5562013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5562407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5562618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5562860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5563081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5563322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5563715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5564104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5564345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.5564581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.5564969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5565362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5565654Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:40:54.5565882Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:40:54.5565982Z ok (22.283s)
2023-01-11T21:40:54.5566002Z 
2023-01-11T21:40:54.5566267Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5566381Z Ran 1 test in 22.283s
2023-01-11T21:40:54.5566399Z 
2023-01-11T21:40:54.5566490Z OK
2023-01-11T21:40:54.5566510Z 
2023-01-11T21:40:54.5566689Z Generating XML reports...
2023-01-11T21:40:54.5567152Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213429.xml
2023-01-11T21:40:54.5567504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5567677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5568062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5568251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5568271Z 
2023-01-11T21:40:54.5568378Z Running tests...
2023-01-11T21:40:54.5568641Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5568952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5569368Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.5569388Z 
2023-01-11T21:40:54.5569646Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5569740Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5569759Z 
2023-01-11T21:40:54.5569865Z OK (skipped=1)
2023-01-11T21:40:54.5569884Z 
2023-01-11T21:40:54.5570009Z Generating XML reports...
2023-01-11T21:40:54.5570454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213454.xml
2023-01-11T21:40:54.5570821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5570995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5571371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5571564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5571583Z 
2023-01-11T21:40:54.5571690Z Running tests...
2023-01-11T21:40:54.5571936Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5572245Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5572633Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.5572656Z 
2023-01-11T21:40:54.5572914Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5573024Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5573043Z 
2023-01-11T21:40:54.5573148Z OK (skipped=1)
2023-01-11T21:40:54.5573167Z 
2023-01-11T21:40:54.5573288Z Generating XML reports...
2023-01-11T21:40:54.5573736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml
2023-01-11T21:40:54.5574088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5574261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5574636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5574881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5574900Z 
2023-01-11T21:40:54.5575007Z Running tests...
2023-01-11T21:40:54.5575266Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5575575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5575993Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.5576013Z 
2023-01-11T21:40:54.5576317Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5576416Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5576435Z 
2023-01-11T21:40:54.5576541Z OK (skipped=1)
2023-01-11T21:40:54.5576560Z 
2023-01-11T21:40:54.5576683Z Generating XML reports...
2023-01-11T21:40:54.5577132Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213459.xml
2023-01-11T21:40:54.5577502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5577674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5578051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5578239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5578259Z 
2023-01-11T21:40:54.5578366Z Running tests...
2023-01-11T21:40:54.5578614Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5578924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5579335Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.5579355Z 
2023-01-11T21:40:54.5579616Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5579724Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5579744Z 
2023-01-11T21:40:54.5579848Z OK (skipped=1)
2023-01-11T21:40:54.5579867Z 
2023-01-11T21:40:54.5579987Z Generating XML reports...
2023-01-11T21:40:54.5580432Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213501.xml
2023-01-11T21:40:54.5580800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5580959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5581335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5581523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5581542Z 
2023-01-11T21:40:54.5581647Z Running tests...
2023-01-11T21:40:54.5581908Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5582216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5582622Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.5582642Z 
2023-01-11T21:40:54.5582893Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5582988Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5583022Z 
2023-01-11T21:40:54.5583114Z OK (skipped=1)
2023-01-11T21:40:54.5583134Z 
2023-01-11T21:40:54.5583254Z Generating XML reports...
2023-01-11T21:40:54.5583699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213504.xml
2023-01-11T21:40:54.5584064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5584294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5584673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5584862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5584882Z 
2023-01-11T21:40:54.5584991Z Running tests...
2023-01-11T21:40:54.5585236Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5585603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5585881Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5586098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16928
2023-01-11T21:40:54.5586312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16929
2023-01-11T21:40:54.5586682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5586858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5587234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5587421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5587767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5587941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5588311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5588497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5588742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5588988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5589387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5589779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5589995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5590225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5590325Z ok (5.860s)
2023-01-11T21:40:54.5590345Z 
2023-01-11T21:40:54.5590605Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5590714Z Ran 1 test in 5.860s
2023-01-11T21:40:54.5590733Z 
2023-01-11T21:40:54.5590823Z OK
2023-01-11T21:40:54.5590845Z 
2023-01-11T21:40:54.5590968Z Generating XML reports...
2023-01-11T21:40:54.5591417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213506.xml
2023-01-11T21:40:54.5591789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5591948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5592330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5592520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5592539Z 
2023-01-11T21:40:54.5592645Z Running tests...
2023-01-11T21:40:54.5592906Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5593215Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5593546Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5593765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17042
2023-01-11T21:40:54.5593963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17043
2023-01-11T21:40:54.5594337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5594510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5594932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5595127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5595490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5595660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5596036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5596223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5596452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5596695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5597095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5597487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5597716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5597943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5598047Z ok (5.849s)
2023-01-11T21:40:54.5598067Z 
2023-01-11T21:40:54.5598328Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5598421Z Ran 1 test in 5.849s
2023-01-11T21:40:54.5598458Z 
2023-01-11T21:40:54.5598532Z OK
2023-01-11T21:40:54.5598551Z 
2023-01-11T21:40:54.5598671Z Generating XML reports...
2023-01-11T21:40:54.5599117Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213515.xml
2023-01-11T21:40:54.5599707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5599888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5600272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5600461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5600481Z 
2023-01-11T21:40:54.5600587Z Running tests...
2023-01-11T21:40:54.5600833Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5601147Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5601419Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5601638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17156
2023-01-11T21:40:54.5601856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17157
2023-01-11T21:40:54.5602227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5602399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5602775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5603047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5603398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5603569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5603939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5604177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5604429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5604670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5605068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5605464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5605676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5605903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5606004Z ok (6.941s)
2023-01-11T21:40:54.5606024Z 
2023-01-11T21:40:54.5606289Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5606399Z Ran 1 test in 6.941s
2023-01-11T21:40:54.5606418Z 
2023-01-11T21:40:54.5606509Z OK
2023-01-11T21:40:54.5606528Z 
2023-01-11T21:40:54.5606649Z Generating XML reports...
2023-01-11T21:40:54.5607097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213523.xml
2023-01-11T21:40:54.5607464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5607627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5608004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5608193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5608212Z 
2023-01-11T21:40:54.5608318Z Running tests...
2023-01-11T21:40:54.5608580Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5608893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5609157Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5609373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17271
2023-01-11T21:40:54.5609572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17272
2023-01-11T21:40:54.5609941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5610113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5610488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5610675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5611040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5611216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5611582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5611771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5612054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5612301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5612702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5613098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5613411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5613643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5613744Z ok (5.832s)
2023-01-11T21:40:54.5613765Z 
2023-01-11T21:40:54.5614028Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5614127Z Ran 1 test in 5.833s
2023-01-11T21:40:54.5614163Z 
2023-01-11T21:40:54.5614238Z OK
2023-01-11T21:40:54.5614257Z 
2023-01-11T21:40:54.5614380Z Generating XML reports...
2023-01-11T21:40:54.5614826Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213533.xml
2023-01-11T21:40:54.5615193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5615367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5615746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5615934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5615954Z 
2023-01-11T21:40:54.5616060Z Running tests...
2023-01-11T21:40:54.5616302Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5616618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5616887Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5617637Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.617s)
2023-01-11T21:40:54.5617657Z 
2023-01-11T21:40:54.5617916Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5618027Z Ran 1 test in 1.618s
2023-01-11T21:40:54.5618046Z 
2023-01-11T21:40:54.5618153Z OK (skipped=1)
2023-01-11T21:40:54.5618172Z 
2023-01-11T21:40:54.5618293Z Generating XML reports...
2023-01-11T21:40:54.5618739Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213541.xml
2023-01-11T21:40:54.5619111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5619268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5619649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5619837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5619856Z 
2023-01-11T21:40:54.5619961Z Running tests...
2023-01-11T21:40:54.5620223Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5620533Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5620782Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:40:54.5620802Z 
2023-01-11T21:40:54.5621056Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5621205Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5621241Z 
2023-01-11T21:40:54.5621332Z OK (skipped=1)
2023-01-11T21:40:54.5621351Z 
2023-01-11T21:40:54.5621471Z Generating XML reports...
2023-01-11T21:40:54.5621917Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213545.xml
2023-01-11T21:40:54.5622285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5622502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5622886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5623076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5623096Z 
2023-01-11T21:40:54.5623203Z Running tests...
2023-01-11T21:40:54.5623446Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5623764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5624036Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:40:54.5624056Z 
2023-01-11T21:40:54.5624314Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5624423Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5624442Z 
2023-01-11T21:40:54.5624548Z OK (skipped=1)
2023-01-11T21:40:54.5624570Z 
2023-01-11T21:40:54.5624692Z Generating XML reports...
2023-01-11T21:40:54.5625140Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213548.xml
2023-01-11T21:40:54.5625508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5625664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5626047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5626235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5626255Z 
2023-01-11T21:40:54.5626362Z Running tests...
2023-01-11T21:40:54.5626621Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5626932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5627245Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:40:54.5627265Z 
2023-01-11T21:40:54.5627522Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5627631Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5627650Z 
2023-01-11T21:40:54.5627740Z OK (skipped=1)
2023-01-11T21:40:54.5627763Z 
2023-01-11T21:40:54.5627887Z Generating XML reports...
2023-01-11T21:40:54.5628330Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213550.xml
2023-01-11T21:40:54.5628698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5628875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5629252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5629444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5629464Z 
2023-01-11T21:40:54.5629570Z Running tests...
2023-01-11T21:40:54.5629814Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5630127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5630491Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5630711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17518
2023-01-11T21:40:54.5630925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17519
2023-01-11T21:40:54.5631297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5631515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5631901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5632091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5632435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5632614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5632984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5633171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5633415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5633658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5634060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5634458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5634686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5634896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5634997Z ok (4.226s)
2023-01-11T21:40:54.5635017Z 
2023-01-11T21:40:54.5635278Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5635390Z Ran 1 test in 4.226s
2023-01-11T21:40:54.5635410Z 
2023-01-11T21:40:54.5635500Z OK
2023-01-11T21:40:54.5635520Z 
2023-01-11T21:40:54.5635642Z Generating XML reports...
2023-01-11T21:40:54.5636091Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213552.xml
2023-01-11T21:40:54.5636458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5636616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5636996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5637187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5637207Z 
2023-01-11T21:40:54.5637313Z Running tests...
2023-01-11T21:40:54.5637574Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5637885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5638176Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5638395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17623
2023-01-11T21:40:54.5638613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17624
2023-01-11T21:40:54.5638967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5639140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5639792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5639987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5640355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5640527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5640967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5641158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5641386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5641629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5642028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5642431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5642659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5642886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5642988Z ok (4.349s)
2023-01-11T21:40:54.5643007Z 
2023-01-11T21:40:54.5643273Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5643384Z Ran 1 test in 4.349s
2023-01-11T21:40:54.5643404Z 
2023-01-11T21:40:54.5643478Z OK
2023-01-11T21:40:54.5643496Z 
2023-01-11T21:40:54.5643620Z Generating XML reports...
2023-01-11T21:40:54.5644070Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213559.xml
2023-01-11T21:40:54.5644443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5644616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5644995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5645185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5645204Z 
2023-01-11T21:40:54.5645312Z Running tests...
2023-01-11T21:40:54.5645573Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5645873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5646145Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:40:54.5646165Z 
2023-01-11T21:40:54.5646422Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5646535Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5646554Z 
2023-01-11T21:40:54.5646660Z OK (skipped=1)
2023-01-11T21:40:54.5646679Z 
2023-01-11T21:40:54.5646800Z Generating XML reports...
2023-01-11T21:40:54.5647244Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213606.xml
2023-01-11T21:40:54.5647611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5647771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5648150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5648339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5648358Z 
2023-01-11T21:40:54.5648465Z Running tests...
2023-01-11T21:40:54.5648722Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5649104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5649404Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:40:54.5649424Z 
2023-01-11T21:40:54.5649676Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5649785Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5649805Z 
2023-01-11T21:40:54.5649952Z OK (skipped=1)
2023-01-11T21:40:54.5649990Z 
2023-01-11T21:40:54.5650102Z Generating XML reports...
2023-01-11T21:40:54.5650550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213609.xml
2023-01-11T21:40:54.5650920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5651096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5651472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5651660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5651680Z 
2023-01-11T21:40:54.5651786Z Running tests...
2023-01-11T21:40:54.5652044Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5652342Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5652617Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5653361Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.619s)
2023-01-11T21:40:54.5653386Z 
2023-01-11T21:40:54.5653644Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5653755Z Ran 1 test in 1.619s
2023-01-11T21:40:54.5653775Z 
2023-01-11T21:40:54.5653879Z OK (skipped=1)
2023-01-11T21:40:54.5653898Z 
2023-01-11T21:40:54.5654022Z Generating XML reports...
2023-01-11T21:40:54.5654464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213611.xml
2023-01-11T21:40:54.5654836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5655010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5655372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5655558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5655581Z 
2023-01-11T21:40:54.5655687Z Running tests...
2023-01-11T21:40:54.5655952Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5656263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5656544Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5656763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17828
2023-01-11T21:40:54.5656982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17829
2023-01-11T21:40:54.5657333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5657507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5657887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5658133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5658498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5658670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5659039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5659269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5659522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5659747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5660148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5660546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5660774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5661002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5661103Z ok (7.455s)
2023-01-11T21:40:54.5661123Z 
2023-01-11T21:40:54.5661383Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5661499Z Ran 1 test in 7.455s
2023-01-11T21:40:54.5661519Z 
2023-01-11T21:40:54.5661594Z OK
2023-01-11T21:40:54.5661630Z 
2023-01-11T21:40:54.5661737Z Generating XML reports...
2023-01-11T21:40:54.5662187Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213615.xml
2023-01-11T21:40:54.5662554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5662731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5663110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5663300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5663320Z 
2023-01-11T21:40:54.5663427Z Running tests...
2023-01-11T21:40:54.5663683Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5663982Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5664248Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5664465Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17947
2023-01-11T21:40:54.5664681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17948
2023-01-11T21:40:54.5665051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5665224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5665599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5665787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5666139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5666312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5666683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5666870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5667173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5667415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5667816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5668206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5668476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5668693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5668797Z ok (7.056s)
2023-01-11T21:40:54.5668817Z 
2023-01-11T21:40:54.5669082Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5669192Z Ran 1 test in 7.057s
2023-01-11T21:40:54.5669216Z 
2023-01-11T21:40:54.5669308Z OK
2023-01-11T21:40:54.5669327Z 
2023-01-11T21:40:54.5669449Z Generating XML reports...
2023-01-11T21:40:54.5669897Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213625.xml
2023-01-11T21:40:54.5670262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5670436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5670801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5670988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5671008Z 
2023-01-11T21:40:54.5671114Z Running tests...
2023-01-11T21:40:54.5671371Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5671683Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5671972Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5672189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18062
2023-01-11T21:40:54.5672408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18063
2023-01-11T21:40:54.5672756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5672933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5673308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5673497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5673857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5674032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5674402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5674586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5674832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5675060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5675460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5675854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5676082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5676366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5676468Z ok (6.826s)
2023-01-11T21:40:54.5676488Z 
2023-01-11T21:40:54.5676750Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5676859Z Ran 1 test in 6.826s
2023-01-11T21:40:54.5676880Z 
2023-01-11T21:40:54.5676953Z OK
2023-01-11T21:40:54.5676990Z 
2023-01-11T21:40:54.5677096Z Generating XML reports...
2023-01-11T21:40:54.5677588Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213635.xml
2023-01-11T21:40:54.5677970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5678144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5678521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5678713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5678732Z 
2023-01-11T21:40:54.5678839Z Running tests...
2023-01-11T21:40:54.5679098Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5679603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5679895Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5680655Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2023-01-11T21:40:54.5680676Z 
2023-01-11T21:40:54.5680936Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5681050Z Ran 1 test in 1.634s
2023-01-11T21:40:54.5681069Z 
2023-01-11T21:40:54.5681175Z OK (skipped=1)
2023-01-11T21:40:54.5681194Z 
2023-01-11T21:40:54.5681316Z Generating XML reports...
2023-01-11T21:40:54.5681760Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213644.xml
2023-01-11T21:40:54.5682128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5682306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5682670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5682861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5682880Z 
2023-01-11T21:40:54.5682986Z Running tests...
2023-01-11T21:40:54.5683246Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5683561Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5683856Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5684600Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2023-01-11T21:40:54.5684621Z 
2023-01-11T21:40:54.5684880Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5684991Z Ran 1 test in 1.645s
2023-01-11T21:40:54.5685010Z 
2023-01-11T21:40:54.5685099Z OK (skipped=1)
2023-01-11T21:40:54.5685135Z 
2023-01-11T21:40:54.5685242Z Generating XML reports...
2023-01-11T21:40:54.5685685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213648.xml
2023-01-11T21:40:54.5686132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5686306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5686683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5686926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5686947Z 
2023-01-11T21:40:54.5687059Z Running tests...
2023-01-11T21:40:54.5687321Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5687618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5687926Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5688148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18245
2023-01-11T21:40:54.5688367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18246
2023-01-11T21:40:54.5688734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5688912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5689293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5689481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5689832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5690004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5690383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5690567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5690811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5691053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5691455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5691851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5692077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5692289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5692442Z skip: Need at least 4 CUDA devices (4.246s)
2023-01-11T21:40:54.5692463Z 
2023-01-11T21:40:54.5692725Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5692835Z Ran 1 test in 4.246s
2023-01-11T21:40:54.5692854Z 
2023-01-11T21:40:54.5692959Z OK (skipped=1)
2023-01-11T21:40:54.5692978Z 
2023-01-11T21:40:54.5693101Z Generating XML reports...
2023-01-11T21:40:54.5693549Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213652.xml
2023-01-11T21:40:54.5693918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5694094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5694454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5694646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5694719Z 
2023-01-11T21:40:54.5694833Z Running tests...
2023-01-11T21:40:54.5695095Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5695407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5695730Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5695990Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18350
2023-01-11T21:40:54.5696212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18351
2023-01-11T21:40:54.5696583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5696739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5697114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5697307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5697669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5697840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5698207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5698398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5698641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5698868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5699268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5699665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5699894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5700122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5700272Z skip: Need at least 4 CUDA devices (4.245s)
2023-01-11T21:40:54.5700292Z 
2023-01-11T21:40:54.5700554Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5700666Z Ran 1 test in 4.245s
2023-01-11T21:40:54.5700686Z 
2023-01-11T21:40:54.5700792Z OK (skipped=1)
2023-01-11T21:40:54.5700811Z 
2023-01-11T21:40:54.5700917Z Generating XML reports...
2023-01-11T21:40:54.5701370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213659.xml
2023-01-11T21:40:54.5701741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5701914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5702289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5702479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5702498Z 
2023-01-11T21:40:54.5702607Z Running tests...
2023-01-11T21:40:54.5702870Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5703165Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5703449Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5704191Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2023-01-11T21:40:54.5704292Z 
2023-01-11T21:40:54.5704560Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5704672Z Ran 1 test in 1.630s
2023-01-11T21:40:54.5704692Z 
2023-01-11T21:40:54.5704797Z OK (skipped=1)
2023-01-11T21:40:54.5704816Z 
2023-01-11T21:40:54.5704935Z Generating XML reports...
2023-01-11T21:40:54.5705420Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213706.xml
2023-01-11T21:40:54.5705796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5705972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5706330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5706525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5706545Z 
2023-01-11T21:40:54.5706651Z Running tests...
2023-01-11T21:40:54.5706911Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5707220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5707491Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5707511Z 
2023-01-11T21:40:54.5707769Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5707879Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5707898Z 
2023-01-11T21:40:54.5708004Z OK (skipped=1)
2023-01-11T21:40:54.5708023Z 
2023-01-11T21:40:54.5708128Z Generating XML reports...
2023-01-11T21:40:54.5708568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213710.xml
2023-01-11T21:40:54.5708939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5709112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5709486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5709677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5709696Z 
2023-01-11T21:40:54.5709803Z Running tests...
2023-01-11T21:40:54.5710062Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5710373Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5710624Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5710646Z 
2023-01-11T21:40:54.5710903Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5711013Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5711032Z 
2023-01-11T21:40:54.5711138Z OK (skipped=1)
2023-01-11T21:40:54.5711157Z 
2023-01-11T21:40:54.5711278Z Generating XML reports...
2023-01-11T21:40:54.5711724Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213713.xml
2023-01-11T21:40:54.5712096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5712270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5712629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5712820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5712889Z 
2023-01-11T21:40:54.5713001Z Running tests...
2023-01-11T21:40:54.5713306Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5713616Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5713890Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5713909Z 
2023-01-11T21:40:54.5714164Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5714334Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5714355Z 
2023-01-11T21:40:54.5714467Z OK (skipped=1)
2023-01-11T21:40:54.5714487Z 
2023-01-11T21:40:54.5714593Z Generating XML reports...
2023-01-11T21:40:54.5715038Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213715.xml
2023-01-11T21:40:54.5715404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5715581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5715956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5716144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5716163Z 
2023-01-11T21:40:54.5716269Z Running tests...
2023-01-11T21:40:54.5716526Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5716839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5717090Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5717109Z 
2023-01-11T21:40:54.5717368Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5717478Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5717500Z 
2023-01-11T21:40:54.5717605Z OK (skipped=1)
2023-01-11T21:40:54.5717624Z 
2023-01-11T21:40:54.5717744Z Generating XML reports...
2023-01-11T21:40:54.5718191Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213717.xml
2023-01-11T21:40:54.5718557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5718729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5719110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5719284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5719304Z 
2023-01-11T21:40:54.5719622Z Running tests...
2023-01-11T21:40:54.5719895Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5720207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5720473Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5720493Z 
2023-01-11T21:40:54.5720749Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5720857Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5720876Z 
2023-01-11T21:40:54.5720982Z OK (skipped=1)
2023-01-11T21:40:54.5721001Z 
2023-01-11T21:40:54.5721107Z Generating XML reports...
2023-01-11T21:40:54.5721558Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213720.xml
2023-01-11T21:40:54.5721928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5722101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5722479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5722746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5722766Z 
2023-01-11T21:40:54.5722873Z Running tests...
2023-01-11T21:40:54.5723136Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5723449Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5723750Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5723790Z 
2023-01-11T21:40:54.5724038Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5724148Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5724167Z 
2023-01-11T21:40:54.5724275Z OK (skipped=1)
2023-01-11T21:40:54.5724294Z 
2023-01-11T21:40:54.5724415Z Generating XML reports...
2023-01-11T21:40:54.5724864Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213722.xml
2023-01-11T21:40:54.5725237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5725410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5725791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5725967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5725987Z 
2023-01-11T21:40:54.5726098Z Running tests...
2023-01-11T21:40:54.5726355Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5726668Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5726937Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5726960Z 
2023-01-11T21:40:54.5727217Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5727328Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5727347Z 
2023-01-11T21:40:54.5727453Z OK (skipped=1)
2023-01-11T21:40:54.5727473Z 
2023-01-11T21:40:54.5727597Z Generating XML reports...
2023-01-11T21:40:54.5728023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213725.xml
2023-01-11T21:40:54.5728392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5728564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5728939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5729129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5729148Z 
2023-01-11T21:40:54.5729257Z Running tests...
2023-01-11T21:40:54.5729512Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5729824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5730068Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5730104Z 
2023-01-11T21:40:54.5730343Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5730454Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5730473Z 
2023-01-11T21:40:54.5730582Z OK (skipped=1)
2023-01-11T21:40:54.5730601Z 
2023-01-11T21:40:54.5730723Z Generating XML reports...
2023-01-11T21:40:54.5731168Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213727.xml
2023-01-11T21:40:54.5731535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5731767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5732148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5732321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5732340Z 
2023-01-11T21:40:54.5732448Z Running tests...
2023-01-11T21:40:54.5732708Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5733061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5733322Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5733341Z 
2023-01-11T21:40:54.5733601Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5733711Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5733730Z 
2023-01-11T21:40:54.5733835Z OK (skipped=1)
2023-01-11T21:40:54.5733858Z 
2023-01-11T21:40:54.5733978Z Generating XML reports...
2023-01-11T21:40:54.5734410Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml
2023-01-11T21:40:54.5734778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5734949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5735329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5735517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5735536Z 
2023-01-11T21:40:54.5735642Z Running tests...
2023-01-11T21:40:54.5735901Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5736211Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5736452Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5736489Z 
2023-01-11T21:40:54.5736730Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5736841Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5736860Z 
2023-01-11T21:40:54.5736966Z OK (skipped=1)
2023-01-11T21:40:54.5736985Z 
2023-01-11T21:40:54.5737106Z Generating XML reports...
2023-01-11T21:40:54.5737553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213732.xml
2023-01-11T21:40:54.5737923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5738099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5738475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5738650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5738686Z 
2023-01-11T21:40:54.5738778Z Running tests...
2023-01-11T21:40:54.5739039Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5739347Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5739600Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5739819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18819
2023-01-11T21:40:54.5740033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18820
2023-01-11T21:40:54.5740399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5740555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5740927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5741164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5741519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5741685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5742045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5742263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5742504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5742736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5743120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5743518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5743746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5743971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5744310Z STAGE:2023-01-11 21:37:40 18820:18820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5745072Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:40:54.5745186Z   warnings.warn(
2023-01-11T21:40:54.5745516Z STAGE:2023-01-11 21:37:40 18819:18819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5746282Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:40:54.5746392Z   warnings.warn(
2023-01-11T21:40:54.5746710Z STAGE:2023-01-11 21:37:40 18819:18819 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5747048Z STAGE:2023-01-11 21:37:40 18820:18820 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5747393Z STAGE:2023-01-11 21:37:40 18820:18820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5747737Z STAGE:2023-01-11 21:37:40 18819:18819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5748060Z STAGE:2023-01-11 21:37:40 18819:18819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5748393Z STAGE:2023-01-11 21:37:40 18819:18819 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5748733Z STAGE:2023-01-11 21:37:40 18819:18819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5749052Z STAGE:2023-01-11 21:37:40 18820:18820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5749381Z STAGE:2023-01-11 21:37:40 18820:18820 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5749705Z STAGE:2023-01-11 21:37:40 18820:18820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5749805Z ok (6.468s)
2023-01-11T21:40:54.5749824Z 
2023-01-11T21:40:54.5750086Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5750196Z Ran 1 test in 6.468s
2023-01-11T21:40:54.5750216Z 
2023-01-11T21:40:54.5750363Z OK
2023-01-11T21:40:54.5750383Z 
2023-01-11T21:40:54.5750507Z Generating XML reports...
2023-01-11T21:40:54.5750957Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213734.xml
2023-01-11T21:40:54.5751325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5751483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5751905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5752104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5752124Z 
2023-01-11T21:40:54.5752230Z Running tests...
2023-01-11T21:40:54.5752489Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5752799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5753063Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5753083Z 
2023-01-11T21:40:54.5753336Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5753447Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5753466Z 
2023-01-11T21:40:54.5753556Z OK (skipped=1)
2023-01-11T21:40:54.5753574Z 
2023-01-11T21:40:54.5753698Z Generating XML reports...
2023-01-11T21:40:54.5754151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213743.xml
2023-01-11T21:40:54.5754520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5754695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5755073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5755267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5755287Z 
2023-01-11T21:40:54.5755394Z Running tests...
2023-01-11T21:40:54.5755656Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5755951Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5756220Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5756441Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18970
2023-01-11T21:40:54.5756657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18971
2023-01-11T21:40:54.5757024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5757198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5757577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5757765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5758109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5758282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5758656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5758845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5759091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5759338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5759965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5760444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5760671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5760883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5761274Z STAGE:2023-01-11 21:37:51 18971:18971 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5761606Z STAGE:2023-01-11 21:37:51 18970:18970 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5761938Z STAGE:2023-01-11 21:37:52 18970:18970 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5762271Z STAGE:2023-01-11 21:37:52 18971:18971 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5762619Z STAGE:2023-01-11 21:37:52 18970:18970 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5762962Z STAGE:2023-01-11 21:37:52 18971:18971 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5763284Z STAGE:2023-01-11 21:37:53 18970:18970 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5763609Z STAGE:2023-01-11 21:37:53 18970:18970 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5763940Z STAGE:2023-01-11 21:37:53 18970:18970 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5764263Z STAGE:2023-01-11 21:37:53 18971:18971 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5764585Z STAGE:2023-01-11 21:37:53 18971:18971 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5764924Z STAGE:2023-01-11 21:37:53 18971:18971 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5765027Z ok (7.869s)
2023-01-11T21:40:54.5765047Z 
2023-01-11T21:40:54.5765308Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5765417Z Ran 1 test in 7.869s
2023-01-11T21:40:54.5765437Z 
2023-01-11T21:40:54.5765527Z OK
2023-01-11T21:40:54.5765546Z 
2023-01-11T21:40:54.5765651Z Generating XML reports...
2023-01-11T21:40:54.5766100Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213746.xml
2023-01-11T21:40:54.5766472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5766769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5767162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5767351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5767375Z 
2023-01-11T21:40:54.5767482Z Running tests...
2023-01-11T21:40:54.5767746Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5768058Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5768307Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5768527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19089
2023-01-11T21:40:54.5768746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19090
2023-01-11T21:40:54.5769311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5769490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5769870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5770128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5770501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5770652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5771028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5771260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5771509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5771752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5772156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5772557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5772786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5773013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5773097Z ok (6.835s)
2023-01-11T21:40:54.5773134Z 
2023-01-11T21:40:54.5773382Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5773544Z Ran 1 test in 6.835s
2023-01-11T21:40:54.5773564Z 
2023-01-11T21:40:54.5773658Z OK
2023-01-11T21:40:54.5773677Z 
2023-01-11T21:40:54.5773800Z Generating XML reports...
2023-01-11T21:40:54.5774251Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213756.xml
2023-01-11T21:40:54.5774619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5774798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5775178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5775351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5775371Z 
2023-01-11T21:40:54.5775478Z Running tests...
2023-01-11T21:40:54.5775738Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5776051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5776306Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5776326Z 
2023-01-11T21:40:54.5776588Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5776699Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5776721Z 
2023-01-11T21:40:54.5776828Z OK (skipped=1)
2023-01-11T21:40:54.5776847Z 
2023-01-11T21:40:54.5776953Z Generating XML reports...
2023-01-11T21:40:54.5777400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213805.xml
2023-01-11T21:40:54.5777766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5777941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5778323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5778512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5778532Z 
2023-01-11T21:40:54.5778638Z Running tests...
2023-01-11T21:40:54.5778898Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5779211Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5779504Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5779724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19237
2023-01-11T21:40:54.5779937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19238
2023-01-11T21:40:54.5780304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5780535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5780922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5781112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5781474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5781634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5782008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5782193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5782436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5782683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5783082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5783474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5783704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5783931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5784252Z STAGE:2023-01-11 21:38:13 19238:19238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5784578Z STAGE:2023-01-11 21:38:13 19237:19237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5784910Z STAGE:2023-01-11 21:38:13 19238:19238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5785238Z STAGE:2023-01-11 21:38:13 19237:19237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5785806Z STAGE:2023-01-11 21:38:13 19237:19237 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:38:13 19238:19238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5785828Z 
2023-01-11T21:40:54.5786156Z STAGE:2023-01-11 21:38:14 19237:19237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5786484Z STAGE:2023-01-11 21:38:14 19238:19238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5786814Z STAGE:2023-01-11 21:38:14 19237:19237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5787159Z STAGE:2023-01-11 21:38:14 19237:19237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5787487Z STAGE:2023-01-11 21:38:14 19238:19238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5787817Z STAGE:2023-01-11 21:38:14 19238:19238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5787920Z ok (6.349s)
2023-01-11T21:40:54.5787939Z 
2023-01-11T21:40:54.5788199Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5788308Z Ran 1 test in 6.350s
2023-01-11T21:40:54.5788328Z 
2023-01-11T21:40:54.5788418Z OK
2023-01-11T21:40:54.5788437Z 
2023-01-11T21:40:54.5788617Z Generating XML reports...
2023-01-11T21:40:54.5789067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213808.xml
2023-01-11T21:40:54.5789438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5789611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5789976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5790209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5790230Z 
2023-01-11T21:40:54.5790341Z Running tests...
2023-01-11T21:40:54.5790601Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5790914Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5791177Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5791398Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19355
2023-01-11T21:40:54.5791612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19356
2023-01-11T21:40:54.5791966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5792141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5792520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5792712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5793076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5793248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5793625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5793811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5794039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5794285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5794688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5795080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5795308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5795532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5795866Z STAGE:2023-01-11 21:38:22 19355:19355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5796195Z STAGE:2023-01-11 21:38:22 19356:19356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5796525Z STAGE:2023-01-11 21:38:22 19356:19356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5796842Z STAGE:2023-01-11 21:38:22 19355:19355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5797189Z STAGE:2023-01-11 21:38:22 19356:19356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5797534Z STAGE:2023-01-11 21:38:22 19355:19355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5797857Z STAGE:2023-01-11 21:38:22 19355:19355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5798185Z STAGE:2023-01-11 21:38:22 19355:19355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5798583Z STAGE:2023-01-11 21:38:22 19355:19355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5798906Z STAGE:2023-01-11 21:38:22 19356:19356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5799233Z STAGE:2023-01-11 21:38:22 19356:19356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5799749Z STAGE:2023-01-11 21:38:22 19356:19356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5799910Z ok (6.414s)
2023-01-11T21:40:54.5799949Z 
2023-01-11T21:40:54.5800208Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5800321Z Ran 1 test in 6.414s
2023-01-11T21:40:54.5800340Z 
2023-01-11T21:40:54.5800431Z OK
2023-01-11T21:40:54.5800450Z 
2023-01-11T21:40:54.5800573Z Generating XML reports...
2023-01-11T21:40:54.5801021Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213817.xml
2023-01-11T21:40:54.5801396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5801569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5801948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5802122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5802146Z 
2023-01-11T21:40:54.5802254Z Running tests...
2023-01-11T21:40:54.5802516Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5802831Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5803094Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5803117Z 
2023-01-11T21:40:54.5803372Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5803481Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5803500Z 
2023-01-11T21:40:54.5803605Z OK (skipped=1)
2023-01-11T21:40:54.5803624Z 
2023-01-11T21:40:54.5803746Z Generating XML reports...
2023-01-11T21:40:54.5804177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213826.xml
2023-01-11T21:40:54.5804548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5804723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5805099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5805287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5805306Z 
2023-01-11T21:40:54.5805418Z Running tests...
2023-01-11T21:40:54.5805676Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5805988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5806223Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5806262Z 
2023-01-11T21:40:54.5806503Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5806613Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5806633Z 
2023-01-11T21:40:54.5806743Z OK (skipped=1)
2023-01-11T21:40:54.5806762Z 
2023-01-11T21:40:54.5806884Z Generating XML reports...
2023-01-11T21:40:54.5807333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213828.xml
2023-01-11T21:40:54.5807699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5807952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5808333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5808505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5808526Z 
2023-01-11T21:40:54.5808634Z Running tests...
2023-01-11T21:40:54.5808894Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5809253Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5809524Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5809544Z 
2023-01-11T21:40:54.5809802Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5809912Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5809932Z 
2023-01-11T21:40:54.5810043Z OK (skipped=1)
2023-01-11T21:40:54.5810062Z 
2023-01-11T21:40:54.5810184Z Generating XML reports...
2023-01-11T21:40:54.5810612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213830.xml
2023-01-11T21:40:54.5810981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5811156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5811536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5811726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5811746Z 
2023-01-11T21:40:54.5811852Z Running tests...
2023-01-11T21:40:54.5812112Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5812424Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5812672Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5812709Z 
2023-01-11T21:40:54.5812949Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5813060Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5813079Z 
2023-01-11T21:40:54.5813187Z OK (skipped=1)
2023-01-11T21:40:54.5813206Z 
2023-01-11T21:40:54.5813369Z Generating XML reports...
2023-01-11T21:40:54.5813821Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213833.xml
2023-01-11T21:40:54.5814191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5814365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5814743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5814920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5814957Z 
2023-01-11T21:40:54.5815049Z Running tests...
2023-01-11T21:40:54.5815310Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5815623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5815872Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5816095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19605
2023-01-11T21:40:54.5816312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19606
2023-01-11T21:40:54.5816681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5816837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5817278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5817464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5817826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5817999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5818411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5818607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5818854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5819097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5819480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5819883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5820112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5820340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5820674Z STAGE:2023-01-11 21:38:41 19605:19605 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5820996Z STAGE:2023-01-11 21:38:41 19606:19606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5821327Z STAGE:2023-01-11 21:38:42 19605:19605 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5821674Z STAGE:2023-01-11 21:38:42 19605:19605 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5822008Z STAGE:2023-01-11 21:38:42 19606:19606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5822333Z STAGE:2023-01-11 21:38:42 19606:19606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5822656Z STAGE:2023-01-11 21:38:42 19605:19605 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5822971Z STAGE:2023-01-11 21:38:42 19606:19606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5823308Z STAGE:2023-01-11 21:38:42 19606:19606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5823651Z STAGE:2023-01-11 21:38:42 19606:19606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5823980Z STAGE:2023-01-11 21:38:42 19605:19605 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5824319Z STAGE:2023-01-11 21:38:42 19605:19605 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5824424Z ok (7.864s)
2023-01-11T21:40:54.5824444Z 
2023-01-11T21:40:54.5824706Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5824800Z Ran 1 test in 7.865s
2023-01-11T21:40:54.5824819Z 
2023-01-11T21:40:54.5824910Z OK
2023-01-11T21:40:54.5824929Z 
2023-01-11T21:40:54.5825050Z Generating XML reports...
2023-01-11T21:40:54.5825498Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213835.xml
2023-01-11T21:40:54.5825865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5826039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5826414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5826602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5826673Z 
2023-01-11T21:40:54.5826784Z Running tests...
2023-01-11T21:40:54.5827030Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5827343Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5827608Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5827826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19728
2023-01-11T21:40:54.5828086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19729
2023-01-11T21:40:54.5828466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5828638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5829002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5829162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5829540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5829728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5830101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5830291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5830538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5830780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5831179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5831579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5831790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5832018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5832353Z STAGE:2023-01-11 21:38:51 19728:19728 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5832678Z STAGE:2023-01-11 21:38:51 19729:19729 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5833010Z STAGE:2023-01-11 21:38:53 19728:19728 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5833355Z STAGE:2023-01-11 21:38:53 19728:19728 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5833689Z STAGE:2023-01-11 21:38:53 19729:19729 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5834037Z STAGE:2023-01-11 21:38:53 19729:19729 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5834346Z STAGE:2023-01-11 21:38:53 19728:19728 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5834664Z STAGE:2023-01-11 21:38:53 19729:19729 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5834992Z STAGE:2023-01-11 21:38:53 19729:19729 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5835335Z STAGE:2023-01-11 21:38:53 19729:19729 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5835658Z STAGE:2023-01-11 21:38:53 19728:19728 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5836000Z STAGE:2023-01-11 21:38:53 19728:19728 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5836101Z ok (7.823s)
2023-01-11T21:40:54.5836121Z 
2023-01-11T21:40:54.5836440Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5836551Z Ran 1 test in 7.824s
2023-01-11T21:40:54.5836571Z 
2023-01-11T21:40:54.5836645Z OK
2023-01-11T21:40:54.5836663Z 
2023-01-11T21:40:54.5836786Z Generating XML reports...
2023-01-11T21:40:54.5837236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213846.xml
2023-01-11T21:40:54.5837602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5837818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5838206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5838397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5838417Z 
2023-01-11T21:40:54.5838524Z Running tests...
2023-01-11T21:40:54.5838787Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5839083Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5839346Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5839366Z 
2023-01-11T21:40:54.5839888Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5839999Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5840019Z 
2023-01-11T21:40:54.5840127Z OK (skipped=1)
2023-01-11T21:40:54.5840151Z 
2023-01-11T21:40:54.5840272Z Generating XML reports...
2023-01-11T21:40:54.5840719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213856.xml
2023-01-11T21:40:54.5841085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5841264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5841623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5841811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5841830Z 
2023-01-11T21:40:54.5841938Z Running tests...
2023-01-11T21:40:54.5842194Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5842503Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5842765Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:40:54.5842785Z 
2023-01-11T21:40:54.5843043Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5843154Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5843173Z 
2023-01-11T21:40:54.5843263Z OK (skipped=1)
2023-01-11T21:40:54.5843302Z 
2023-01-11T21:40:54.5843408Z Generating XML reports...
2023-01-11T21:40:54.5843857Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213858.xml
2023-01-11T21:40:54.5844227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5844399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5844779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5844967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5844987Z 
2023-01-11T21:40:54.5845093Z Running tests...
2023-01-11T21:40:54.5845354Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5845647Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5846125Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:40:54.5846145Z 
2023-01-11T21:40:54.5846402Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5846512Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5846532Z 
2023-01-11T21:40:54.5846636Z OK (skipped=1)
2023-01-11T21:40:54.5846655Z 
2023-01-11T21:40:54.5846777Z Generating XML reports...
2023-01-11T21:40:54.5847323Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213901.xml
2023-01-11T21:40:54.5847706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5847879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5848238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5848432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5848452Z 
2023-01-11T21:40:54.5848557Z Running tests...
2023-01-11T21:40:54.5848814Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5849122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5849383Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:40:54.5849403Z 
2023-01-11T21:40:54.5849663Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5849772Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5849792Z 
2023-01-11T21:40:54.5849880Z OK (skipped=1)
2023-01-11T21:40:54.5849915Z 
2023-01-11T21:40:54.5850022Z Generating XML reports...
2023-01-11T21:40:54.5850466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213903.xml
2023-01-11T21:40:54.5850836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5851010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5851389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5851577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5851597Z 
2023-01-11T21:40:54.5851703Z Running tests...
2023-01-11T21:40:54.5851965Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5852260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5852543Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T21:40:54.5852563Z 
2023-01-11T21:40:54.5852819Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5852931Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5852951Z 
2023-01-11T21:40:54.5853055Z OK (skipped=1)
2023-01-11T21:40:54.5853074Z 
2023-01-11T21:40:54.5853194Z Generating XML reports...
2023-01-11T21:40:54.5853640Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213906.xml
2023-01-11T21:40:54.5854006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5854182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5854545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5854735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5854755Z 
2023-01-11T21:40:54.5854861Z Running tests...
2023-01-11T21:40:54.5855118Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5855490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5855799Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T21:40:54.5855819Z 
2023-01-11T21:40:54.5856076Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5856186Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5856205Z 
2023-01-11T21:40:54.5856353Z OK (skipped=1)
2023-01-11T21:40:54.5856374Z 
2023-01-11T21:40:54.5856484Z Generating XML reports...
2023-01-11T21:40:54.5856930Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213908.xml
2023-01-11T21:40:54.5857295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5857474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5857849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5858038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5858057Z 
2023-01-11T21:40:54.5858164Z Running tests...
2023-01-11T21:40:54.5858421Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5858718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5859023Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T21:40:54.5859043Z 
2023-01-11T21:40:54.5859298Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5859405Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5859425Z 
2023-01-11T21:40:54.5859532Z OK (skipped=1)
2023-01-11T21:40:54.5859551Z 
2023-01-11T21:40:54.5859671Z Generating XML reports...
2023-01-11T21:40:54.5860117Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213910.xml
2023-01-11T21:40:54.5860483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5860652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5861017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5861208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5861227Z 
2023-01-11T21:40:54.5861333Z Running tests...
2023-01-11T21:40:54.5861589Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5861899Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5862189Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:40:54.5862209Z 
2023-01-11T21:40:54.5862465Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5862574Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5862593Z 
2023-01-11T21:40:54.5862698Z OK (skipped=1)
2023-01-11T21:40:54.5862717Z 
2023-01-11T21:40:54.5862822Z Generating XML reports...
2023-01-11T21:40:54.5863272Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213913.xml
2023-01-11T21:40:54.5863635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5863806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5864180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5864423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5864443Z 
2023-01-11T21:40:54.5864550Z Running tests...
2023-01-11T21:40:54.5864809Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5865120Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5865355Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5865617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20115
2023-01-11T21:40:54.5865841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20116
2023-01-11T21:40:54.5866209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5866382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5866762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5866950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5867310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5867464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5867839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5868025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5868273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5868517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5868919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5869314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5869540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5869766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5869850Z ok (5.852s)
2023-01-11T21:40:54.5869870Z 
2023-01-11T21:40:54.5870133Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5870245Z Ran 1 test in 5.852s
2023-01-11T21:40:54.5870264Z 
2023-01-11T21:40:54.5870354Z OK
2023-01-11T21:40:54.5870373Z 
2023-01-11T21:40:54.5870496Z Generating XML reports...
2023-01-11T21:40:54.5870946Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213915.xml
2023-01-11T21:40:54.5871318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5871491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5871854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5872044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5872063Z 
2023-01-11T21:40:54.5872170Z Running tests...
2023-01-11T21:40:54.5872432Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5872744Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5873023Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5873241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20233
2023-01-11T21:40:54.5873522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20234
2023-01-11T21:40:54.5873891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5874047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5874424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5874658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5875031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5875205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5875574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5875761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5876004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5876243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5876625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5877022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5877250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5877474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5877809Z STAGE:2023-01-11 21:39:29 20234:20234 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5878133Z STAGE:2023-01-11 21:39:29 20233:20233 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5878464Z STAGE:2023-01-11 21:39:29 20234:20234 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5878788Z STAGE:2023-01-11 21:39:29 20233:20233 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5879131Z STAGE:2023-01-11 21:39:29 20234:20234 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5879638Z STAGE:2023-01-11 21:39:29 20233:20233 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5879745Z ok (6.442s)
2023-01-11T21:40:54.5879765Z 
2023-01-11T21:40:54.5880031Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5880141Z Ran 1 test in 6.442s
2023-01-11T21:40:54.5880161Z 
2023-01-11T21:40:54.5880252Z OK
2023-01-11T21:40:54.5880271Z 
2023-01-11T21:40:54.5880393Z Generating XML reports...
2023-01-11T21:40:54.5880844Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213924.xml
2023-01-11T21:40:54.5881210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5881367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5881745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5881939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5881959Z 
2023-01-11T21:40:54.5882065Z Running tests...
2023-01-11T21:40:54.5882324Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5882636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5882909Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5883206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20355
2023-01-11T21:40:54.5883405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20356
2023-01-11T21:40:54.5883772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5883944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5884374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5884572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5884936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5885106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5885479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5885667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5885896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5886140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5886533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5886927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5887153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5887378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5887713Z STAGE:2023-01-11 21:39:38 20355:20355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5888033Z STAGE:2023-01-11 21:39:38 20356:20356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:40:54.5888363Z STAGE:2023-01-11 21:39:38 20356:20356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5888678Z STAGE:2023-01-11 21:39:38 20355:20355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:40:54.5889027Z STAGE:2023-01-11 21:39:38 20356:20356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5889373Z STAGE:2023-01-11 21:39:38 20355:20355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:40:54.5889472Z ok (6.434s)
2023-01-11T21:40:54.5889492Z 
2023-01-11T21:40:54.5889750Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5889857Z Ran 1 test in 6.434s
2023-01-11T21:40:54.5889881Z 
2023-01-11T21:40:54.5889969Z OK
2023-01-11T21:40:54.5889989Z 
2023-01-11T21:40:54.5890107Z Generating XML reports...
2023-01-11T21:40:54.5890541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213933.xml
2023-01-11T21:40:54.5890906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5891079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5891457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5891646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5891666Z 
2023-01-11T21:40:54.5891770Z Running tests...
2023-01-11T21:40:54.5892029Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5892341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5892683Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:40:54.5892703Z 
2023-01-11T21:40:54.5892962Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5893057Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5893076Z 
2023-01-11T21:40:54.5893182Z OK (skipped=1)
2023-01-11T21:40:54.5893201Z 
2023-01-11T21:40:54.5893323Z Generating XML reports...
2023-01-11T21:40:54.5893811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213942.xml
2023-01-11T21:40:54.5894187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5894361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5894736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5894925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5894944Z 
2023-01-11T21:40:54.5895036Z Running tests...
2023-01-11T21:40:54.5895294Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5895597Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5895867Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:40:54.5895887Z 
2023-01-11T21:40:54.5896141Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5896251Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5896270Z 
2023-01-11T21:40:54.5896375Z OK (skipped=1)
2023-01-11T21:40:54.5896394Z 
2023-01-11T21:40:54.5896517Z Generating XML reports...
2023-01-11T21:40:54.5896963Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213944.xml
2023-01-11T21:40:54.5897321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5897493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5897870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5898062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5898082Z 
2023-01-11T21:40:54.5898186Z Running tests...
2023-01-11T21:40:54.5898445Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5898752Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5899048Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:40:54.5899072Z 
2023-01-11T21:40:54.5899330Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5899425Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5899444Z 
2023-01-11T21:40:54.5899547Z OK (skipped=1)
2023-01-11T21:40:54.5899566Z 
2023-01-11T21:40:54.5899686Z Generating XML reports...
2023-01-11T21:40:54.5900127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213946.xml
2023-01-11T21:40:54.5900493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5900666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5901043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5901228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5901302Z 
2023-01-11T21:40:54.5901414Z Running tests...
2023-01-11T21:40:54.5901658Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5901965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5902257Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:40:54.5902279Z 
2023-01-11T21:40:54.5902579Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5902694Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5902714Z 
2023-01-11T21:40:54.5902819Z OK (skipped=1)
2023-01-11T21:40:54.5902838Z 
2023-01-11T21:40:54.5902959Z Generating XML reports...
2023-01-11T21:40:54.5903405Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213949.xml
2023-01-11T21:40:54.5903759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5903938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5904316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5904501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5904521Z 
2023-01-11T21:40:54.5904625Z Running tests...
2023-01-11T21:40:54.5904883Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5905194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5905476Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T21:40:54.5905496Z 
2023-01-11T21:40:54.5905751Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5905849Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5905869Z 
2023-01-11T21:40:54.5905977Z OK (skipped=1)
2023-01-11T21:40:54.5905996Z 
2023-01-11T21:40:54.5906117Z Generating XML reports...
2023-01-11T21:40:54.5906559Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213951.xml
2023-01-11T21:40:54.5906926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5907097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5907476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5907665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5907684Z 
2023-01-11T21:40:54.5907789Z Running tests...
2023-01-11T21:40:54.5908032Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5908346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5908630Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T21:40:54.5908649Z 
2023-01-11T21:40:54.5908904Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5909013Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5909032Z 
2023-01-11T21:40:54.5909134Z OK (skipped=1)
2023-01-11T21:40:54.5909153Z 
2023-01-11T21:40:54.5909277Z Generating XML reports...
2023-01-11T21:40:54.5909719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213954.xml
2023-01-11T21:40:54.5910083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5910240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5910674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5910860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5910880Z 
2023-01-11T21:40:54.5910983Z Running tests...
2023-01-11T21:40:54.5911236Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5911544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5911864Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5912088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20675
2023-01-11T21:40:54.5912287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20676
2023-01-11T21:40:54.5912655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5912830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5913205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5913436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5913800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5913970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5914343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5914531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5914761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5915002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5915403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5915791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5916018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5916245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5916345Z ok (7.477s)
2023-01-11T21:40:54.5916365Z 
2023-01-11T21:40:54.5916624Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5916717Z Ran 1 test in 7.478s
2023-01-11T21:40:54.5916753Z 
2023-01-11T21:40:54.5916829Z OK
2023-01-11T21:40:54.5916848Z 
2023-01-11T21:40:54.5916967Z Generating XML reports...
2023-01-11T21:40:54.5917418Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213956.xml
2023-01-11T21:40:54.5917785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5917959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5918336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5918525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5918545Z 
2023-01-11T21:40:54.5918651Z Running tests...
2023-01-11T21:40:54.5918894Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5919202Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5919638Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T21:40:54.5919732Z 
2023-01-11T21:40:54.5920003Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5920112Z Ran 1 test in 0.002s
2023-01-11T21:40:54.5920132Z 
2023-01-11T21:40:54.5920237Z OK (skipped=1)
2023-01-11T21:40:54.5920256Z 
2023-01-11T21:40:54.5920377Z Generating XML reports...
2023-01-11T21:40:54.5920826Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214006.xml
2023-01-11T21:40:54.5921256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5921421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5921800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5921989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5922012Z 
2023-01-11T21:40:54.5922117Z Running tests...
2023-01-11T21:40:54.5922380Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5922689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5922938Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5923156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20827
2023-01-11T21:40:54.5923358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20828
2023-01-11T21:40:54.5923726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5923896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5924269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5924461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5924821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5924991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5925365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5925551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5925781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5926022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5926419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5926815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5927042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5927267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5927367Z ok (6.944s)
2023-01-11T21:40:54.5927386Z 
2023-01-11T21:40:54.5927647Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5927743Z Ran 1 test in 6.944s
2023-01-11T21:40:54.5927781Z 
2023-01-11T21:40:54.5927857Z OK
2023-01-11T21:40:54.5927876Z 
2023-01-11T21:40:54.5927996Z Generating XML reports...
2023-01-11T21:40:54.5928442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214008.xml
2023-01-11T21:40:54.5928808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5929039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5929419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5929606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5929626Z 
2023-01-11T21:40:54.5929734Z Running tests...
2023-01-11T21:40:54.5929976Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5930327Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5930626Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5930843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20942
2023-01-11T21:40:54.5931055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20943
2023-01-11T21:40:54.5931426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5931596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5931966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5932135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5932493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5932659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5933026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5933208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5933450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5933691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5934088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5934482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5934698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5934926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5935714Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:40:54.5936506Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:40:54.5936610Z ok (7.354s)
2023-01-11T21:40:54.5936631Z 
2023-01-11T21:40:54.5936893Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5937056Z Ran 1 test in 7.354s
2023-01-11T21:40:54.5937076Z 
2023-01-11T21:40:54.5937165Z OK
2023-01-11T21:40:54.5937185Z 
2023-01-11T21:40:54.5937305Z Generating XML reports...
2023-01-11T21:40:54.5937756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214018.xml
2023-01-11T21:40:54.5938124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5938298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5938700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5938896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5938916Z 
2023-01-11T21:40:54.5939023Z Running tests...
2023-01-11T21:40:54.5939279Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5939596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5939876Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5940090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21061
2023-01-11T21:40:54.5940299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21062
2023-01-11T21:40:54.5940673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5940831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5941209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5941393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5941757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5941931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5942302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5942489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5942732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5942962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5943360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5943753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5943985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5944212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5944450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5944689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5945088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5945478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5945713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.5945930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.5946376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5946765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5946867Z ok (11.449s)
2023-01-11T21:40:54.5946887Z 
2023-01-11T21:40:54.5947146Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5947257Z Ran 1 test in 11.449s
2023-01-11T21:40:54.5947276Z 
2023-01-11T21:40:54.5947405Z OK
2023-01-11T21:40:54.5947426Z 
2023-01-11T21:40:54.5947551Z Generating XML reports...
2023-01-11T21:40:54.5947985Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214028.xml
2023-01-11T21:40:54.5948356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5948535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5948913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5949102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5949122Z 
2023-01-11T21:40:54.5949227Z Running tests...
2023-01-11T21:40:54.5949491Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5949803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:40:54.5950090Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:40:54.5950293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21183
2023-01-11T21:40:54.5950509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21184
2023-01-11T21:40:54.5950877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5951053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5951428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5951612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5951970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:40:54.5952143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:40:54.5952497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:40:54.5952681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:40:54.5952923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:40:54.5953165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:40:54.5953565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5953955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:40:54.5954179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:40:54.5954406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:40:54.5954645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:40:54.5954868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:40:54.5955266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5955714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:40:54.5955951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:40:54.5956183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:40:54.5956618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5957008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:40:54.5957107Z ok (11.329s)
2023-01-11T21:40:54.5957126Z 
2023-01-11T21:40:54.5957382Z ----------------------------------------------------------------------
2023-01-11T21:40:54.5957481Z Ran 1 test in 11.329s
2023-01-11T21:40:54.5957514Z 
2023-01-11T21:40:54.5957591Z OK
2023-01-11T21:40:54.5957610Z 
2023-01-11T21:40:54.5957726Z Generating XML reports...
2023-01-11T21:40:54.5958170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214042.xml
2023-01-11T21:40:54.5958189Z 
2023-01-11T21:40:54.5958619Z ##[endgroup]
2023-01-11T21:40:54.5959089Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_wk42wko9)
2023-01-11T21:40:54.5959114Z 
2023-01-11T21:40:54.5959322Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2023-01-11T21:40:54.5960021Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:40:54.347293]
2023-01-11T22:07:08.9035544Z 
2023-01-11T22:07:08.9038708Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T22:07:08.9040785Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_s9pi1hvh)
2023-01-11T22:07:08.9041640Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:07:08.9093591Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T22:07:08.9154758Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9158748Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9159933Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9160710Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9161451Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9162309Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9163180Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9164021Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9164884Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9165873Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9166883Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9167816Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9168783Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9169712Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9170586Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9171408Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9172214Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9172963Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9173734Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9174550Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9175473Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9176344Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9177108Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9178071Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9178910Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9179705Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9180480Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9181253Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9182125Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9182830Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9183509Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9184249Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9184987Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9185631Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9186430Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9187215Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9187937Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9188679Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9189404Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9190101Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9190833Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9191606Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9192428Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9193212Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9194006Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9194791Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9195553Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9196298Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9197063Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9197839Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9198641Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9199368Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9200580Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9201370Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9202157Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9203003Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9203821Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9204556Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9204969Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9205360Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9205745Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9206143Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9206521Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9206894Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9207261Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9207782Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9208185Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9208560Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9208944Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9209324Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9209695Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9210151Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9210560Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9210969Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9211337Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9211710Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9212089Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9212459Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9212851Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9213251Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9213625Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9214013Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9214420Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9214848Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9215267Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9215709Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9216159Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9216603Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9217049Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9217491Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9217924Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9218343Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9218785Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9219229Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9219667Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9220137Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9220588Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9221037Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9221443Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9221832Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9222210Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9222554Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9222916Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9223295Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9223669Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9224053Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9224433Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9224899Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9225288Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9225683Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9226075Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9226458Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9226880Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9227338Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9227760Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9228157Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9228565Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9228989Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9229399Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9229807Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9230193Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9230543Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9230928Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9231315Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9231703Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9232080Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9232529Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9233034Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9233485Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9233941Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9234413Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9234840Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9235231Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9235650Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9236080Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9236508Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9236973Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9237410Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9237818Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9238254Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9238658Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9239028Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9239795Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9240314Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9240728Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9241141Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9241577Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9241992Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9242495Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9242936Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9243434Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9243993Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9244645Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9245257Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9245854Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9246461Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9247063Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9247638Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9248228Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9248827Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9249319Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9249743Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9250128Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9250523Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9250912Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9251309Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9251734Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9252162Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9252625Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9253088Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9253502Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9253873Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9254285Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9254720Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9255132Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9255544Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9255955Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9256382Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9256797Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9257216Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9257627Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9258019Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9258507Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9258911Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9259311Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9259769Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9260202Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9260598Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9261003Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9261408Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9261828Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9262225Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9262602Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9262969Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9263321Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9263698Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9264073Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9264441Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9264807Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9265187Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9265549Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9265887Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9266261Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9266654Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9267028Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9267402Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9267747Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9268123Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9268506Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9268916Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9269366Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9269799Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9270211Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9270630Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9271042Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9271468Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9271892Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9272305Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9272700Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9273104Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9273511Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9273891Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9274288Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9274751Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9275236Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9275689Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9276203Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9276660Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9277097Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9277528Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9277947Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9278417Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9278845Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9279287Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9280316Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9280820Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9281302Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9281720Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9282110Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9282493Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9282898Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9283285Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9283647Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9284025Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9284410Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9284759Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9285123Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9285495Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9285870Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9286248Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9286652Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9287026Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9287378Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9287761Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9288141Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9288485Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9288847Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9289221Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9289578Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9289958Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9290350Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9290727Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9291089Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9291455Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9291831Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9292233Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9292672Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9293099Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9293472Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9293973Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9294397Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9294803Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9295179Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9295586Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9296079Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9296483Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9296883Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9297287Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9297684Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9298054Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9298471Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9298916Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9299339Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9300082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9300537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9301116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9301564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9301795Z 
2023-01-11T22:07:08.9301904Z Running tests...
2023-01-11T22:07:08.9302311Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9302829Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9303414Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9303970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21338
2023-01-11T22:07:08.9304410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21339
2023-01-11T22:07:08.9305003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9305446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9306019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9306464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9307043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9307481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9308048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9308487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9308941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9309432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9310086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9310756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9311339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9311806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9312308Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9313199Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9313866Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9314683Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9315324Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9316149Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9316797Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9317609Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9318262Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9319065Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9320244Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9321080Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9321734Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9322532Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9323178Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:07:08.9323984Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:07:08.9324462Z ok (6.944s)
2023-01-11T22:07:08.9324601Z 
2023-01-11T22:07:08.9324869Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9325192Z Ran 1 test in 6.944s
2023-01-11T22:07:08.9325352Z 
2023-01-11T22:07:08.9325444Z OK
2023-01-11T22:07:08.9325577Z 
2023-01-11T22:07:08.9325684Z Generating XML reports...
2023-01-11T22:07:08.9326287Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214058.xml
2023-01-11T22:07:08.9326999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9327447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9328001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9328462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9328688Z 
2023-01-11T22:07:08.9328894Z Running tests...
2023-01-11T22:07:08.9329283Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9329808Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9330332Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T22:07:08.9330637Z 
2023-01-11T22:07:08.9330901Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9331207Z Ran 1 test in 0.004s
2023-01-11T22:07:08.9331364Z 
2023-01-11T22:07:08.9331547Z OK (skipped=1)
2023-01-11T22:07:08.9331712Z 
2023-01-11T22:07:08.9331840Z Generating XML reports...
2023-01-11T22:07:08.9332428Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214108.xml
2023-01-11T22:07:08.9333141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9333593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9334167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9334614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9334841Z 
2023-01-11T22:07:08.9334948Z Running tests...
2023-01-11T22:07:08.9335345Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9335876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9336362Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9336846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21486
2023-01-11T22:07:08.9337352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21487
2023-01-11T22:07:08.9337946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9338391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9338955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9339417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9339976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9340414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9340980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9341423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9341871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9342365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9343014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9343680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9344202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9344670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9345011Z ok (4.229s)
2023-01-11T22:07:08.9345142Z 
2023-01-11T22:07:08.9345409Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9345732Z Ran 1 test in 4.229s
2023-01-11T22:07:08.9345888Z 
2023-01-11T22:07:08.9345979Z OK
2023-01-11T22:07:08.9346182Z 
2023-01-11T22:07:08.9346292Z Generating XML reports...
2023-01-11T22:07:08.9346896Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214110.xml
2023-01-11T22:07:08.9347603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9348045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9348699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9349170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9349394Z 
2023-01-11T22:07:08.9349502Z Running tests...
2023-01-11T22:07:08.9349886Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9350406Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9350936Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9351987Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2023-01-11T22:07:08.9352502Z 
2023-01-11T22:07:08.9352763Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9353073Z Ran 1 test in 1.635s
2023-01-11T22:07:08.9353232Z 
2023-01-11T22:07:08.9353339Z OK (skipped=1)
2023-01-11T22:07:08.9353489Z 
2023-01-11T22:07:08.9353611Z Generating XML reports...
2023-01-11T22:07:08.9354194Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214117.xml
2023-01-11T22:07:08.9354900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9355349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9355916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9356361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9356584Z 
2023-01-11T22:07:08.9356695Z Running tests...
2023-01-11T22:07:08.9357093Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9357602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9358149Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T22:07:08.9358472Z 
2023-01-11T22:07:08.9358734Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9359060Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9359218Z 
2023-01-11T22:07:08.9359307Z OK (skipped=1)
2023-01-11T22:07:08.9359646Z 
2023-01-11T22:07:08.9359773Z Generating XML reports...
2023-01-11T22:07:08.9360380Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214121.xml
2023-01-11T22:07:08.9361088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9361515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9362088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9362554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9362781Z 
2023-01-11T22:07:08.9362872Z Running tests...
2023-01-11T22:07:08.9363269Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9363887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9364450Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T22:07:08.9364769Z 
2023-01-11T22:07:08.9365029Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9365344Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9365503Z 
2023-01-11T22:07:08.9365610Z OK (skipped=1)
2023-01-11T22:07:08.9365761Z 
2023-01-11T22:07:08.9365924Z Generating XML reports...
2023-01-11T22:07:08.9366537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214123.xml
2023-01-11T22:07:08.9367239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9367684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9368241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9368700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9368924Z 
2023-01-11T22:07:08.9369030Z Running tests...
2023-01-11T22:07:08.9369409Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9369937Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9370480Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9371003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21691
2023-01-11T22:07:08.9371430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21692
2023-01-11T22:07:08.9372026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9372473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9373040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9373489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9374060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9374503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9375052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9375512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9375962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9376452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9377094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9377774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9378287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9378757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9379216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9379693Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9380166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9380619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9381152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9381621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9382085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9382538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9383044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9383520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9383968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9384433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9384776Z ok (7.973s)
2023-01-11T22:07:08.9384920Z 
2023-01-11T22:07:08.9385196Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9385501Z Ran 1 test in 7.973s
2023-01-11T22:07:08.9385661Z 
2023-01-11T22:07:08.9385752Z OK
2023-01-11T22:07:08.9385884Z 
2023-01-11T22:07:08.9386006Z Generating XML reports...
2023-01-11T22:07:08.9386594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214126.xml
2023-01-11T22:07:08.9387301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9387748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9388320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9388772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9389003Z 
2023-01-11T22:07:08.9389110Z Running tests...
2023-01-11T22:07:08.9389508Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9390016Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9390570Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9391098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21810
2023-01-11T22:07:08.9391547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21811
2023-01-11T22:07:08.9392141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9392584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9393154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9393624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9394179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9394617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9395252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9395698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9396146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9396634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9397287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9398019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9398538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9399005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9399721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9400263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9400751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9401226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9401551Z ok (6.920s)
2023-01-11T22:07:08.9401699Z 
2023-01-11T22:07:08.9401974Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9402304Z Ran 1 test in 6.920s
2023-01-11T22:07:08.9402463Z 
2023-01-11T22:07:08.9402537Z OK
2023-01-11T22:07:08.9402670Z 
2023-01-11T22:07:08.9402793Z Generating XML reports...
2023-01-11T22:07:08.9403398Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214136.xml
2023-01-11T22:07:08.9404108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9404533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9405104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9405568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9405795Z 
2023-01-11T22:07:08.9405904Z Running tests...
2023-01-11T22:07:08.9406286Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9406815Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9407378Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9407903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21929
2023-01-11T22:07:08.9408347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21930
2023-01-11T22:07:08.9408953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9409396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9409947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9410411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9410989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9411409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9411979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9412438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9412883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9413362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9414017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9414699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9415297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9415746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9416209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9416694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9417148Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9417665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9418142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9418607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9419058Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9419524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9419865Z ok (6.938s)
2023-01-11T22:07:08.9420010Z 
2023-01-11T22:07:08.9420269Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9420594Z Ran 1 test in 6.939s
2023-01-11T22:07:08.9420751Z 
2023-01-11T22:07:08.9420844Z OK
2023-01-11T22:07:08.9420975Z 
2023-01-11T22:07:08.9421098Z Generating XML reports...
2023-01-11T22:07:08.9421690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214146.xml
2023-01-11T22:07:08.9422406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9422852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9423404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9423877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9424103Z 
2023-01-11T22:07:08.9424211Z Running tests...
2023-01-11T22:07:08.9424608Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9425115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9425700Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9426258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22048
2023-01-11T22:07:08.9426681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22049
2023-01-11T22:07:08.9427285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9427729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9428304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9428755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9429324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9429759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9430332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9430778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9431229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9431721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9432425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9433109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9433628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9434093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9434605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9435099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9435444Z ok (7.050s)
2023-01-11T22:07:08.9435592Z 
2023-01-11T22:07:08.9435861Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9436168Z Ran 1 test in 7.050s
2023-01-11T22:07:08.9436330Z 
2023-01-11T22:07:08.9436421Z OK
2023-01-11T22:07:08.9436552Z 
2023-01-11T22:07:08.9436675Z Generating XML reports...
2023-01-11T22:07:08.9437266Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214155.xml
2023-01-11T22:07:08.9437973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9438418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9438991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9439617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9439853Z 
2023-01-11T22:07:08.9439962Z Running tests...
2023-01-11T22:07:08.9440366Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9440870Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9441455Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9442014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22167
2023-01-11T22:07:08.9442460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22168
2023-01-11T22:07:08.9443046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9443491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9444059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9444511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9445086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9445528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9446095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9446536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9446986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9447478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9448128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9448843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9449356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9449916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9450375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9450855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9451325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9451853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9452190Z ok (7.718s)
2023-01-11T22:07:08.9452336Z 
2023-01-11T22:07:08.9452608Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9452930Z Ran 1 test in 7.719s
2023-01-11T22:07:08.9453089Z 
2023-01-11T22:07:08.9453164Z OK
2023-01-11T22:07:08.9453296Z 
2023-01-11T22:07:08.9453419Z Generating XML reports...
2023-01-11T22:07:08.9454028Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214205.xml
2023-01-11T22:07:08.9454739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9455164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9455733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9456196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9456420Z 
2023-01-11T22:07:08.9456512Z Running tests...
2023-01-11T22:07:08.9456909Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9457431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9457991Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9458512Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22286
2023-01-11T22:07:08.9458950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22287
2023-01-11T22:07:08.9459554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9459996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9460553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9461016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9461586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9462006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9462573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9463030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9463475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9463946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9464598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9465287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9465789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9466258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9466792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9467272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9467730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9468195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9468539Z ok (7.425s)
2023-01-11T22:07:08.9468685Z 
2023-01-11T22:07:08.9469005Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9469319Z Ran 1 test in 7.425s
2023-01-11T22:07:08.9469479Z 
2023-01-11T22:07:08.9469571Z OK
2023-01-11T22:07:08.9469704Z 
2023-01-11T22:07:08.9469826Z Generating XML reports...
2023-01-11T22:07:08.9470417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214215.xml
2023-01-11T22:07:08.9471132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9471576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9472143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9472589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9472815Z 
2023-01-11T22:07:08.9472923Z Running tests...
2023-01-11T22:07:08.9473326Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9473835Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9474417Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9474970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22405
2023-01-11T22:07:08.9475417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22406
2023-01-11T22:07:08.9476002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9476445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9477009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9477460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9478029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9478466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9479031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9479658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9480115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9480605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9481262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9481932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9482444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9482908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9483364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9483929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9484401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9484872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9485197Z ok (6.836s)
2023-01-11T22:07:08.9485343Z 
2023-01-11T22:07:08.9485613Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9485936Z Ran 1 test in 6.837s
2023-01-11T22:07:08.9486096Z 
2023-01-11T22:07:08.9486231Z OK
2023-01-11T22:07:08.9486373Z 
2023-01-11T22:07:08.9486498Z Generating XML reports...
2023-01-11T22:07:08.9487105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214225.xml
2023-01-11T22:07:08.9487814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9488245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9488811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9489272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9489495Z 
2023-01-11T22:07:08.9489587Z Running tests...
2023-01-11T22:07:08.9489986Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9490511Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9491059Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9492102Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2023-01-11T22:07:08.9492622Z 
2023-01-11T22:07:08.9492885Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9493206Z Ran 1 test in 1.628s
2023-01-11T22:07:08.9493363Z 
2023-01-11T22:07:08.9493469Z OK (skipped=1)
2023-01-11T22:07:08.9493621Z 
2023-01-11T22:07:08.9493727Z Generating XML reports...
2023-01-11T22:07:08.9494325Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214234.xml
2023-01-11T22:07:08.9495033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9495478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9496035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9496496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9496723Z 
2023-01-11T22:07:08.9496831Z Running tests...
2023-01-11T22:07:08.9497214Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9497736Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9498275Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9498800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22558
2023-01-11T22:07:08.9499228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22559
2023-01-11T22:07:08.9499827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9500272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9500822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9501352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9501929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9502366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9502916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9503417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9503874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9504362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9505003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9505695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9506213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9506662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9507004Z ok (4.331s)
2023-01-11T22:07:08.9507150Z 
2023-01-11T22:07:08.9507419Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9507743Z Ran 1 test in 4.331s
2023-01-11T22:07:08.9507885Z 
2023-01-11T22:07:08.9507978Z OK
2023-01-11T22:07:08.9508112Z 
2023-01-11T22:07:08.9508235Z Generating XML reports...
2023-01-11T22:07:08.9508838Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214239.xml
2023-01-11T22:07:08.9509532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9509985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9510556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9511017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9511225Z 
2023-01-11T22:07:08.9511335Z Running tests...
2023-01-11T22:07:08.9511736Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9512264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9512806Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9513868Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2023-01-11T22:07:08.9514390Z 
2023-01-11T22:07:08.9514654Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9542361Z Ran 1 test in 1.627s
2023-01-11T22:07:08.9542526Z 
2023-01-11T22:07:08.9542645Z OK (skipped=1)
2023-01-11T22:07:08.9542800Z 
2023-01-11T22:07:08.9542927Z Generating XML reports...
2023-01-11T22:07:08.9543595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214245.xml
2023-01-11T22:07:08.9544303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9544756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9545331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9545960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9546172Z 
2023-01-11T22:07:08.9546289Z Running tests...
2023-01-11T22:07:08.9546691Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9547216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9547793Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9548309Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22697
2023-01-11T22:07:08.9548816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22698
2023-01-11T22:07:08.9549425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9549853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9550430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9550893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9551448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9551886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9552454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9552916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9553348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9553844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9554503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9555187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9555687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9556156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9556498Z ok (6.930s)
2023-01-11T22:07:08.9556646Z 
2023-01-11T22:07:08.9556899Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9557227Z Ran 1 test in 6.930s
2023-01-11T22:07:08.9557390Z 
2023-01-11T22:07:08.9557479Z OK
2023-01-11T22:07:08.9557614Z 
2023-01-11T22:07:08.9557740Z Generating XML reports...
2023-01-11T22:07:08.9558327Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214250.xml
2023-01-11T22:07:08.9559039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9559802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9560381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9560848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9561073Z 
2023-01-11T22:07:08.9561186Z Running tests...
2023-01-11T22:07:08.9561585Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9562094Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9562594Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2023-01-11T22:07:08.9562877Z 
2023-01-11T22:07:08.9563237Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9563543Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9563703Z 
2023-01-11T22:07:08.9563811Z OK (skipped=1)
2023-01-11T22:07:08.9563963Z 
2023-01-11T22:07:08.9564086Z Generating XML reports...
2023-01-11T22:07:08.9564689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214259.xml
2023-01-11T22:07:08.9565440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9565894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9566466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9566915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9567143Z 
2023-01-11T22:07:08.9567251Z Running tests...
2023-01-11T22:07:08.9567659Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9568182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9568618Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9569091Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9569567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22845
2023-01-11T22:07:08.9570001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22846
2023-01-11T22:07:08.9570607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9571050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9571618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9572070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9572638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9573073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9573636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9574082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9574529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9575025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9575660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9576344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9576861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9577325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9577650Z ok (7.353s)
2023-01-11T22:07:08.9577798Z 
2023-01-11T22:07:08.9578063Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9578390Z Ran 1 test in 7.353s
2023-01-11T22:07:08.9578548Z 
2023-01-11T22:07:08.9578624Z OK
2023-01-11T22:07:08.9578757Z 
2023-01-11T22:07:08.9578881Z Generating XML reports...
2023-01-11T22:07:08.9579484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214302.xml
2023-01-11T22:07:08.9580191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9580682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9581252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9581714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9581940Z 
2023-01-11T22:07:08.9582049Z Running tests...
2023-01-11T22:07:08.9582431Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9583016Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9583501Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9583976Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9584455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22964
2023-01-11T22:07:08.9584904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22965
2023-01-11T22:07:08.9585508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9585935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9586502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9586967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9587520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9587957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9588521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9588984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9589413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9589910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9590557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9591243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9591742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9592210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9592680Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9593143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9593484Z ok (7.455s)
2023-01-11T22:07:08.9593632Z 
2023-01-11T22:07:08.9593901Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9594222Z Ran 1 test in 7.455s
2023-01-11T22:07:08.9594367Z 
2023-01-11T22:07:08.9594459Z OK
2023-01-11T22:07:08.9594591Z 
2023-01-11T22:07:08.9594715Z Generating XML reports...
2023-01-11T22:07:08.9595324Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214311.xml
2023-01-11T22:07:08.9596020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9596464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9597030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9597556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9597765Z 
2023-01-11T22:07:08.9597873Z Running tests...
2023-01-11T22:07:08.9598272Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9598791Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9599263Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9600122Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9600640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23083
2023-01-11T22:07:08.9601086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23084
2023-01-11T22:07:08.9601685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9602131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9602699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9603146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9603722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9604160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9604732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9605173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9605623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9606118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9606774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9607443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9607960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9608431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9608890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9609371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:08.9609720Z ok (7.334s)
2023-01-11T22:07:08.9609867Z 
2023-01-11T22:07:08.9610135Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9610445Z Ran 1 test in 7.334s
2023-01-11T22:07:08.9610604Z 
2023-01-11T22:07:08.9610696Z OK
2023-01-11T22:07:08.9610828Z 
2023-01-11T22:07:08.9610951Z Generating XML reports...
2023-01-11T22:07:08.9611536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214321.xml
2023-01-11T22:07:08.9612244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9612688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9613260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9613707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9613934Z 
2023-01-11T22:07:08.9614043Z Running tests...
2023-01-11T22:07:08.9614442Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9615033Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9615504Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:07:08.9615990Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9616462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23202
2023-01-11T22:07:08.9616890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23203
2023-01-11T22:07:08.9617536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9617991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9618550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9619015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9619592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9620031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9620577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9621034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9621491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9621987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9622623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9623299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9623816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9624264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9624604Z ok (7.312s)
2023-01-11T22:07:08.9624750Z 
2023-01-11T22:07:08.9625016Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9625338Z Ran 1 test in 7.313s
2023-01-11T22:07:08.9625481Z 
2023-01-11T22:07:08.9625574Z OK
2023-01-11T22:07:08.9625710Z 
2023-01-11T22:07:08.9625836Z Generating XML reports...
2023-01-11T22:07:08.9626438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214331.xml
2023-01-11T22:07:08.9627123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9627567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9628139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9628598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9628821Z 
2023-01-11T22:07:08.9628913Z Running tests...
2023-01-11T22:07:08.9629307Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9629826Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9630307Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9630590Z 
2023-01-11T22:07:08.9630853Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9631171Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9631325Z 
2023-01-11T22:07:08.9631429Z OK (skipped=1)
2023-01-11T22:07:08.9631583Z 
2023-01-11T22:07:08.9631753Z Generating XML reports...
2023-01-11T22:07:08.9632357Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214341.xml
2023-01-11T22:07:08.9633064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9633506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9634062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9634577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9634809Z 
2023-01-11T22:07:08.9634918Z Running tests...
2023-01-11T22:07:08.9635304Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9635827Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9636365Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:07:08.9636685Z 
2023-01-11T22:07:08.9636946Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9637309Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9637465Z 
2023-01-11T22:07:08.9637574Z OK (skipped=1)
2023-01-11T22:07:08.9637728Z 
2023-01-11T22:07:08.9637851Z Generating XML reports...
2023-01-11T22:07:08.9638440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214344.xml
2023-01-11T22:07:08.9639150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9639818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9640398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9640855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9641080Z 
2023-01-11T22:07:08.9641189Z Running tests...
2023-01-11T22:07:08.9641589Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9642099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9642639Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:07:08.9642959Z 
2023-01-11T22:07:08.9643225Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9643545Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9643704Z 
2023-01-11T22:07:08.9643794Z OK (skipped=1)
2023-01-11T22:07:08.9643945Z 
2023-01-11T22:07:08.9644067Z Generating XML reports...
2023-01-11T22:07:08.9644666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214346.xml
2023-01-11T22:07:08.9645374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9645802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9646368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9646827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9647050Z 
2023-01-11T22:07:08.9647141Z Running tests...
2023-01-11T22:07:08.9647540Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9648062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9648633Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:07:08.9648927Z 
2023-01-11T22:07:08.9649190Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9649609Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9649767Z 
2023-01-11T22:07:08.9649873Z OK (skipped=1)
2023-01-11T22:07:08.9650025Z 
2023-01-11T22:07:08.9650131Z Generating XML reports...
2023-01-11T22:07:08.9650736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214348.xml
2023-01-11T22:07:08.9651438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9651940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9652507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9652970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9653194Z 
2023-01-11T22:07:08.9653301Z Running tests...
2023-01-11T22:07:08.9653684Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9654207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9654736Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:07:08.9655052Z 
2023-01-11T22:07:08.9655313Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9655613Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9655770Z 
2023-01-11T22:07:08.9655876Z OK (skipped=1)
2023-01-11T22:07:08.9656033Z 
2023-01-11T22:07:08.9656157Z Generating XML reports...
2023-01-11T22:07:08.9656754Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214351.xml
2023-01-11T22:07:08.9657440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9657887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9658459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9658909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9659135Z 
2023-01-11T22:07:08.9659243Z Running tests...
2023-01-11T22:07:08.9659640Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9660160Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9660682Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:07:08.9661000Z 
2023-01-11T22:07:08.9661260Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9661581Z Ran 1 test in 0.003s
2023-01-11T22:07:08.9661740Z 
2023-01-11T22:07:08.9661831Z OK (skipped=1)
2023-01-11T22:07:08.9661990Z 
2023-01-11T22:07:08.9662115Z Generating XML reports...
2023-01-11T22:07:08.9662712Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214353.xml
2023-01-11T22:07:08.9663416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9663844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9664415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9664874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9665096Z 
2023-01-11T22:07:08.9665205Z Running tests...
2023-01-11T22:07:08.9665585Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9666104Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9666675Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9666966Z 
2023-01-11T22:07:08.9667210Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9667528Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9667688Z 
2023-01-11T22:07:08.9667796Z OK (skipped=1)
2023-01-11T22:07:08.9667947Z 
2023-01-11T22:07:08.9668069Z Generating XML reports...
2023-01-11T22:07:08.9668699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214356.xml
2023-01-11T22:07:08.9669417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9669864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9670416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9670887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9671114Z 
2023-01-11T22:07:08.9671222Z Running tests...
2023-01-11T22:07:08.9671619Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9672122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9672621Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9673106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23552
2023-01-11T22:07:08.9673540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23553
2023-01-11T22:07:08.9674144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9674587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9675164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9675615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9676185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9676620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9677190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9677631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9678081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9678572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9679207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9680150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9680665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9681140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9681704Z STAGE:2023-01-11 21:44:04 23552:23552 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9682268Z STAGE:2023-01-11 21:44:04 23553:23553 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9682839Z STAGE:2023-01-11 21:44:05 23552:23552 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9683416Z STAGE:2023-01-11 21:44:05 23553:23553 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9684070Z STAGE:2023-01-11 21:44:05 23552:23552 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9684667Z STAGE:2023-01-11 21:44:05 23553:23553 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9685237Z STAGE:2023-01-11 21:44:05 23552:23552 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9685790Z STAGE:2023-01-11 21:44:05 23552:23552 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9686437Z STAGE:2023-01-11 21:44:05 23552:23552 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9687028Z STAGE:2023-01-11 21:44:05 23553:23553 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9687590Z STAGE:2023-01-11 21:44:05 23553:23553 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9688155Z STAGE:2023-01-11 21:44:05 23553:23553 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9688516Z ok (7.779s)
2023-01-11T22:07:08.9688663Z 
2023-01-11T22:07:08.9688925Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9689248Z Ran 1 test in 7.779s
2023-01-11T22:07:08.9689392Z 
2023-01-11T22:07:08.9689484Z OK
2023-01-11T22:07:08.9689616Z 
2023-01-11T22:07:08.9689744Z Generating XML reports...
2023-01-11T22:07:08.9690201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214358.xml
2023-01-11T22:07:08.9690573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9690748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9691112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9691302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9691326Z 
2023-01-11T22:07:08.9691433Z Running tests...
2023-01-11T22:07:08.9691696Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9692010Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9692276Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9692495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23671
2023-01-11T22:07:08.9692715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23672
2023-01-11T22:07:08.9693082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9693239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9693620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9693812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9694176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9694346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9694723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9694916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9695163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9695390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9695792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9696254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9696483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9696707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9697040Z STAGE:2023-01-11 21:44:14 23671:23671 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9697413Z STAGE:2023-01-11 21:44:14 23672:23672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9697759Z STAGE:2023-01-11 21:44:15 23671:23671 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9698089Z STAGE:2023-01-11 21:44:15 23672:23672 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9698418Z STAGE:2023-01-11 21:44:15 23671:23671 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9698769Z STAGE:2023-01-11 21:44:15 23672:23672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9699091Z STAGE:2023-01-11 21:44:15 23672:23672 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9699410Z STAGE:2023-01-11 21:44:15 23671:23671 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9699741Z STAGE:2023-01-11 21:44:15 23672:23672 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9700089Z STAGE:2023-01-11 21:44:15 23672:23672 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9700417Z STAGE:2023-01-11 21:44:15 23671:23671 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9700760Z STAGE:2023-01-11 21:44:15 23671:23671 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9700863Z ok (7.843s)
2023-01-11T22:07:08.9700883Z 
2023-01-11T22:07:08.9701133Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9701245Z Ran 1 test in 7.843s
2023-01-11T22:07:08.9701265Z 
2023-01-11T22:07:08.9701356Z OK
2023-01-11T22:07:08.9701375Z 
2023-01-11T22:07:08.9701499Z Generating XML reports...
2023-01-11T22:07:08.9701950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214408.xml
2023-01-11T22:07:08.9702319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9702498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9702879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9703053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9703091Z 
2023-01-11T22:07:08.9703183Z Running tests...
2023-01-11T22:07:08.9703446Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9703760Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9704029Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9704048Z 
2023-01-11T22:07:08.9704304Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9704414Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9704433Z 
2023-01-11T22:07:08.9704544Z OK (skipped=1)
2023-01-11T22:07:08.9704564Z 
2023-01-11T22:07:08.9704686Z Generating XML reports...
2023-01-11T22:07:08.9705114Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214419.xml
2023-01-11T22:07:08.9705481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9705712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9706093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9706282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9706302Z 
2023-01-11T22:07:08.9706409Z Running tests...
2023-01-11T22:07:08.9706665Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9707019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9707295Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9707315Z 
2023-01-11T22:07:08.9707555Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9707666Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9707686Z 
2023-01-11T22:07:08.9707791Z OK (skipped=1)
2023-01-11T22:07:08.9707814Z 
2023-01-11T22:07:08.9707935Z Generating XML reports...
2023-01-11T22:07:08.9708380Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214421.xml
2023-01-11T22:07:08.9708747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9708920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9709298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9709490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9709510Z 
2023-01-11T22:07:08.9709601Z Running tests...
2023-01-11T22:07:08.9709859Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9710170Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9710447Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9710665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23856
2023-01-11T22:07:08.9710882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23857
2023-01-11T22:07:08.9711251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9711426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9711789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9711980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9712339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9712509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9712885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9713073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9713319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9713719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9713965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9714347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9714576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9714802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9715198Z STAGE:2023-01-11 21:44:29 23857:23857 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9715520Z STAGE:2023-01-11 21:44:29 23856:23856 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9715853Z STAGE:2023-01-11 21:44:30 23856:23856 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9716184Z STAGE:2023-01-11 21:44:30 23857:23857 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9716569Z STAGE:2023-01-11 21:44:30 23856:23856 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9716925Z STAGE:2023-01-11 21:44:30 23857:23857 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9717010Z ok (7.737s)
2023-01-11T22:07:08.9717031Z 
2023-01-11T22:07:08.9717292Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9717407Z Ran 1 test in 7.738s
2023-01-11T22:07:08.9717427Z 
2023-01-11T22:07:08.9717519Z OK
2023-01-11T22:07:08.9717537Z 
2023-01-11T22:07:08.9717660Z Generating XML reports...
2023-01-11T22:07:08.9718108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214424.xml
2023-01-11T22:07:08.9718473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9718653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9719016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9719207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9719227Z 
2023-01-11T22:07:08.9719334Z Running tests...
2023-01-11T22:07:08.9719842Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9720166Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9720446Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9720664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23975
2023-01-11T22:07:08.9720882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23976
2023-01-11T22:07:08.9721253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9721410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9721788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9721976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9722336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9722510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9722885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9723071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9723318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9723546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9723948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9724344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9724656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9724881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9725217Z STAGE:2023-01-11 21:44:39 23975:23975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9725537Z STAGE:2023-01-11 21:44:39 23976:23976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9725924Z STAGE:2023-01-11 21:44:41 23975:23975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9726260Z STAGE:2023-01-11 21:44:41 23976:23976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9726588Z STAGE:2023-01-11 21:44:41 23976:23976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9726932Z STAGE:2023-01-11 21:44:41 23975:23975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9727038Z ok (7.656s)
2023-01-11T22:07:08.9727058Z 
2023-01-11T22:07:08.9727323Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9727434Z Ran 1 test in 7.656s
2023-01-11T22:07:08.9727453Z 
2023-01-11T22:07:08.9727545Z OK
2023-01-11T22:07:08.9727564Z 
2023-01-11T22:07:08.9727687Z Generating XML reports...
2023-01-11T22:07:08.9728137Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214434.xml
2023-01-11T22:07:08.9728510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9728670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9729048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9729236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9729260Z 
2023-01-11T22:07:08.9729367Z Running tests...
2023-01-11T22:07:08.9729629Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9729940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9730198Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9730958Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.632s)
2023-01-11T22:07:08.9730980Z 
2023-01-11T22:07:08.9731240Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9731335Z Ran 1 test in 1.632s
2023-01-11T22:07:08.9731371Z 
2023-01-11T22:07:08.9731463Z OK (skipped=1)
2023-01-11T22:07:08.9731486Z 
2023-01-11T22:07:08.9731609Z Generating XML reports...
2023-01-11T22:07:08.9732062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214444.xml
2023-01-11T22:07:08.9732429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9732603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9732986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9733175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9733196Z 
2023-01-11T22:07:08.9733303Z Running tests...
2023-01-11T22:07:08.9733548Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9733858Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9734192Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9734943Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.608s)
2023-01-11T22:07:08.9734964Z 
2023-01-11T22:07:08.9735219Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9735373Z Ran 1 test in 1.608s
2023-01-11T22:07:08.9735395Z 
2023-01-11T22:07:08.9735507Z OK (skipped=1)
2023-01-11T22:07:08.9735527Z 
2023-01-11T22:07:08.9735650Z Generating XML reports...
2023-01-11T22:07:08.9736099Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214448.xml
2023-01-11T22:07:08.9736466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9736631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9737010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9737199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9737218Z 
2023-01-11T22:07:08.9737325Z Running tests...
2023-01-11T22:07:08.9737583Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9737895Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9738171Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9738390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24162
2023-01-11T22:07:08.9738589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24163
2023-01-11T22:07:08.9738966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9739140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9739516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9739706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9740072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9740244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9740613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9740801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9741035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9741277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9741677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9742075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9742309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9742535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9742636Z ok (6.846s)
2023-01-11T22:07:08.9742656Z 
2023-01-11T22:07:08.9742921Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9743016Z Ran 1 test in 6.846s
2023-01-11T22:07:08.9743105Z 
2023-01-11T22:07:08.9743185Z OK
2023-01-11T22:07:08.9743204Z 
2023-01-11T22:07:08.9743329Z Generating XML reports...
2023-01-11T22:07:08.9743786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214452.xml
2023-01-11T22:07:08.9744155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9744328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9744751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9744947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9744967Z 
2023-01-11T22:07:08.9745074Z Running tests...
2023-01-11T22:07:08.9745318Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9745629Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9745903Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9746122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24277
2023-01-11T22:07:08.9746338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24278
2023-01-11T22:07:08.9746709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9746885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9747260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9747432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9747797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9747973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9748344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9748580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9748830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9749078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9749480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9749874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9750087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9750317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9750557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:08.9750798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:08.9751195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:08.9751590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:08.9751829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:08.9752062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:08.9752458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:08.9752898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:08.9753141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T22:07:08.9753376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T22:07:08.9753806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:07:08.9754208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:07:08.9754310Z ok (8.450s)
2023-01-11T22:07:08.9754330Z 
2023-01-11T22:07:08.9754592Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9754707Z Ran 1 test in 8.450s
2023-01-11T22:07:08.9754728Z 
2023-01-11T22:07:08.9754820Z OK
2023-01-11T22:07:08.9754839Z 
2023-01-11T22:07:08.9754945Z Generating XML reports...
2023-01-11T22:07:08.9755396Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214502.xml
2023-01-11T22:07:08.9755766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9755939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9756321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9756511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9756531Z 
2023-01-11T22:07:08.9756639Z Running tests...
2023-01-11T22:07:08.9756899Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9757217Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9757458Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9757674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24428
2023-01-11T22:07:08.9757888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24429
2023-01-11T22:07:08.9758259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9758437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9758816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9759005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9759367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9759839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9760221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9760407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9760653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9760901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9761301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9761697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9761926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9762238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9762323Z ok (6.838s)
2023-01-11T22:07:08.9762343Z 
2023-01-11T22:07:08.9762607Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9762719Z Ran 1 test in 6.838s
2023-01-11T22:07:08.9762738Z 
2023-01-11T22:07:08.9762830Z OK
2023-01-11T22:07:08.9762849Z 
2023-01-11T22:07:08.9762973Z Generating XML reports...
2023-01-11T22:07:08.9763490Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214513.xml
2023-01-11T22:07:08.9763874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9764047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9764406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9764603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9764622Z 
2023-01-11T22:07:08.9764730Z Running tests...
2023-01-11T22:07:08.9764991Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9765302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9765725Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9765745Z 
2023-01-11T22:07:08.9766000Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9766113Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9766133Z 
2023-01-11T22:07:08.9766240Z OK (skipped=1)
2023-01-11T22:07:08.9766259Z 
2023-01-11T22:07:08.9766364Z Generating XML reports...
2023-01-11T22:07:08.9766812Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214522.xml
2023-01-11T22:07:08.9767184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9767358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9767736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9767929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9767953Z 
2023-01-11T22:07:08.9768062Z Running tests...
2023-01-11T22:07:08.9768326Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9768641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9769042Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9769083Z 
2023-01-11T22:07:08.9769320Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9769429Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9769449Z 
2023-01-11T22:07:08.9769556Z OK (skipped=1)
2023-01-11T22:07:08.9769575Z 
2023-01-11T22:07:08.9769697Z Generating XML reports...
2023-01-11T22:07:08.9770142Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214524.xml
2023-01-11T22:07:08.9770511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9770688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9771068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9771242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9771313Z 
2023-01-11T22:07:08.9771427Z Running tests...
2023-01-11T22:07:08.9771688Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9771998Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9772424Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9772444Z 
2023-01-11T22:07:08.9772700Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9772865Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9772886Z 
2023-01-11T22:07:08.9772999Z OK (skipped=1)
2023-01-11T22:07:08.9773018Z 
2023-01-11T22:07:08.9773142Z Generating XML reports...
2023-01-11T22:07:08.9773574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214527.xml
2023-01-11T22:07:08.9773943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9774121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9774499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9774687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9774707Z 
2023-01-11T22:07:08.9774814Z Running tests...
2023-01-11T22:07:08.9775073Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9775387Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9775788Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9775825Z 
2023-01-11T22:07:08.9776064Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9776180Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9776199Z 
2023-01-11T22:07:08.9776306Z OK (skipped=1)
2023-01-11T22:07:08.9776325Z 
2023-01-11T22:07:08.9776445Z Generating XML reports...
2023-01-11T22:07:08.9776893Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214529.xml
2023-01-11T22:07:08.9777261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9777436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9777819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9777992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9778028Z 
2023-01-11T22:07:08.9778120Z Running tests...
2023-01-11T22:07:08.9778381Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9778695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9779105Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9779125Z 
2023-01-11T22:07:08.9779382Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9779493Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9779512Z 
2023-01-11T22:07:08.9779618Z OK (skipped=1)
2023-01-11T22:07:08.9779638Z 
2023-01-11T22:07:08.9779762Z Generating XML reports...
2023-01-11T22:07:08.9780190Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214532.xml
2023-01-11T22:07:08.9780557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9780730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9781167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9781357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9781377Z 
2023-01-11T22:07:08.9781485Z Running tests...
2023-01-11T22:07:08.9781744Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9782056Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9782508Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9782530Z 
2023-01-11T22:07:08.9782777Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9782889Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9782909Z 
2023-01-11T22:07:08.9783015Z OK (skipped=1)
2023-01-11T22:07:08.9783034Z 
2023-01-11T22:07:08.9783161Z Generating XML reports...
2023-01-11T22:07:08.9783608Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214534.xml
2023-01-11T22:07:08.9783974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9784149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9784526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9784722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9784742Z 
2023-01-11T22:07:08.9784833Z Running tests...
2023-01-11T22:07:08.9785093Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9785403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9785822Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9785845Z 
2023-01-11T22:07:08.9786100Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9786211Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9786231Z 
2023-01-11T22:07:08.9786340Z OK (skipped=1)
2023-01-11T22:07:08.9786359Z 
2023-01-11T22:07:08.9786481Z Generating XML reports...
2023-01-11T22:07:08.9786908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214536.xml
2023-01-11T22:07:08.9787281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9787455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9787831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9788023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9788042Z 
2023-01-11T22:07:08.9788149Z Running tests...
2023-01-11T22:07:08.9788408Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9788718Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9789125Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9789145Z 
2023-01-11T22:07:08.9789389Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9789503Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9789523Z 
2023-01-11T22:07:08.9789628Z OK (skipped=1)
2023-01-11T22:07:08.9789647Z 
2023-01-11T22:07:08.9789769Z Generating XML reports...
2023-01-11T22:07:08.9790216Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214539.xml
2023-01-11T22:07:08.9790647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9790820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9791197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9791386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9791406Z 
2023-01-11T22:07:08.9791496Z Running tests...
2023-01-11T22:07:08.9791800Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9792124Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9792520Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9792540Z 
2023-01-11T22:07:08.9792795Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9792909Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9792928Z 
2023-01-11T22:07:08.9793035Z OK (skipped=1)
2023-01-11T22:07:08.9793053Z 
2023-01-11T22:07:08.9793175Z Generating XML reports...
2023-01-11T22:07:08.9793618Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214541.xml
2023-01-11T22:07:08.9793970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9794145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9794520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9794709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9794728Z 
2023-01-11T22:07:08.9794835Z Running tests...
2023-01-11T22:07:08.9795094Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9795409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9795714Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9795735Z 
2023-01-11T22:07:08.9795990Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9796086Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9796105Z 
2023-01-11T22:07:08.9796216Z OK (skipped=1)
2023-01-11T22:07:08.9796238Z 
2023-01-11T22:07:08.9796360Z Generating XML reports...
2023-01-11T22:07:08.9796807Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214544.xml
2023-01-11T22:07:08.9797175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9797352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9797729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9797919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9797938Z 
2023-01-11T22:07:08.9798029Z Running tests...
2023-01-11T22:07:08.9798287Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9798596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9798992Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9799012Z 
2023-01-11T22:07:08.9799266Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9799558Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9799580Z 
2023-01-11T22:07:08.9799696Z OK (skipped=1)
2023-01-11T22:07:08.9799791Z 
2023-01-11T22:07:08.9799920Z Generating XML reports...
2023-01-11T22:07:08.9800375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214546.xml
2023-01-11T22:07:08.9800726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9800902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9801333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9801531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9801551Z 
2023-01-11T22:07:08.9801660Z Running tests...
2023-01-11T22:07:08.9801918Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9802229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9802640Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9802660Z 
2023-01-11T22:07:08.9802920Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9803014Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9803033Z 
2023-01-11T22:07:08.9803139Z OK (skipped=1)
2023-01-11T22:07:08.9803158Z 
2023-01-11T22:07:08.9803280Z Generating XML reports...
2023-01-11T22:07:08.9803731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214548.xml
2023-01-11T22:07:08.9804099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9804274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9804650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9804844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9804863Z 
2023-01-11T22:07:08.9804970Z Running tests...
2023-01-11T22:07:08.9805217Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9805530Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9805928Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:08.9805947Z 
2023-01-11T22:07:08.9806206Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9806316Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9806335Z 
2023-01-11T22:07:08.9806440Z OK (skipped=1)
2023-01-11T22:07:08.9806459Z 
2023-01-11T22:07:08.9806580Z Generating XML reports...
2023-01-11T22:07:08.9807021Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214551.xml
2023-01-11T22:07:08.9807379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9807556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9807933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9808122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9808142Z 
2023-01-11T22:07:08.9808251Z Running tests...
2023-01-11T22:07:08.9808513Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9808824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9809115Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9809186Z 
2023-01-11T22:07:08.9809450Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9809543Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9809579Z 
2023-01-11T22:07:08.9809668Z OK (skipped=1)
2023-01-11T22:07:08.9809687Z 
2023-01-11T22:07:08.9809809Z Generating XML reports...
2023-01-11T22:07:08.9810251Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214553.xml
2023-01-11T22:07:08.9810661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9810842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9811220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9811408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9811428Z 
2023-01-11T22:07:08.9811535Z Running tests...
2023-01-11T22:07:08.9811783Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9812092Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9812364Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9812384Z 
2023-01-11T22:07:08.9812638Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9812749Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9812768Z 
2023-01-11T22:07:08.9812877Z OK (skipped=1)
2023-01-11T22:07:08.9812896Z 
2023-01-11T22:07:08.9813017Z Generating XML reports...
2023-01-11T22:07:08.9813464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214556.xml
2023-01-11T22:07:08.9813834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9813996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9814371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9814560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9814580Z 
2023-01-11T22:07:08.9814688Z Running tests...
2023-01-11T22:07:08.9814945Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9815258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9815529Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9815548Z 
2023-01-11T22:07:08.9815804Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9815897Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9815932Z 
2023-01-11T22:07:08.9816026Z OK (skipped=1)
2023-01-11T22:07:08.9816045Z 
2023-01-11T22:07:08.9816167Z Generating XML reports...
2023-01-11T22:07:08.9816616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214558.xml
2023-01-11T22:07:08.9816986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9817159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9817541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9817731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9817750Z 
2023-01-11T22:07:08.9817857Z Running tests...
2023-01-11T22:07:08.9818100Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9818411Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9818755Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9818775Z 
2023-01-11T22:07:08.9819031Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9819142Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9819162Z 
2023-01-11T22:07:08.9819268Z OK (skipped=1)
2023-01-11T22:07:08.9819287Z 
2023-01-11T22:07:08.9819409Z Generating XML reports...
2023-01-11T22:07:08.9819893Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214600.xml
2023-01-11T22:07:08.9820271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9820428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9820805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9820997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9821016Z 
2023-01-11T22:07:08.9821124Z Running tests...
2023-01-11T22:07:08.9821382Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9821692Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9821962Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9821981Z 
2023-01-11T22:07:08.9822239Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9822351Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9822370Z 
2023-01-11T22:07:08.9822459Z OK (skipped=1)
2023-01-11T22:07:08.9822478Z 
2023-01-11T22:07:08.9822600Z Generating XML reports...
2023-01-11T22:07:08.9823046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214603.xml
2023-01-11T22:07:08.9823419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9823592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9823969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9824158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9824177Z 
2023-01-11T22:07:08.9824289Z Running tests...
2023-01-11T22:07:08.9824532Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9824845Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9825112Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9825131Z 
2023-01-11T22:07:08.9825390Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9825502Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9825521Z 
2023-01-11T22:07:08.9825630Z OK (skipped=1)
2023-01-11T22:07:08.9825649Z 
2023-01-11T22:07:08.9825771Z Generating XML reports...
2023-01-11T22:07:08.9826214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214605.xml
2023-01-11T22:07:08.9826582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9826742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9827118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9827306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9827325Z 
2023-01-11T22:07:08.9827433Z Running tests...
2023-01-11T22:07:08.9827752Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9828064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9828330Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9828350Z 
2023-01-11T22:07:08.9828606Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9828716Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9828735Z 
2023-01-11T22:07:08.9828876Z OK (skipped=1)
2023-01-11T22:07:08.9828897Z 
2023-01-11T22:07:08.9829026Z Generating XML reports...
2023-01-11T22:07:08.9829470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214607.xml
2023-01-11T22:07:08.9829832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9830008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9830378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9830567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9830586Z 
2023-01-11T22:07:08.9830695Z Running tests...
2023-01-11T22:07:08.9830935Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9831250Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9831525Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9831544Z 
2023-01-11T22:07:08.9831798Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9831908Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9831927Z 
2023-01-11T22:07:08.9832038Z OK (skipped=1)
2023-01-11T22:07:08.9832061Z 
2023-01-11T22:07:08.9832184Z Generating XML reports...
2023-01-11T22:07:08.9832630Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214610.xml
2023-01-11T22:07:08.9832997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9833154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9833536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9833726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9833746Z 
2023-01-11T22:07:08.9833853Z Running tests...
2023-01-11T22:07:08.9834112Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9834424Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9834694Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9834714Z 
2023-01-11T22:07:08.9834971Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9835081Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9835100Z 
2023-01-11T22:07:08.9835189Z OK (skipped=1)
2023-01-11T22:07:08.9835208Z 
2023-01-11T22:07:08.9835331Z Generating XML reports...
2023-01-11T22:07:08.9835776Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214612.xml
2023-01-11T22:07:08.9836142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9836315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9836691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9836937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9836957Z 
2023-01-11T22:07:08.9837064Z Running tests...
2023-01-11T22:07:08.9837326Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9837621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9837881Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9837901Z 
2023-01-11T22:07:08.9838204Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9838322Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9838342Z 
2023-01-11T22:07:08.9838448Z OK (skipped=1)
2023-01-11T22:07:08.9838467Z 
2023-01-11T22:07:08.9838588Z Generating XML reports...
2023-01-11T22:07:08.9839036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214615.xml
2023-01-11T22:07:08.9839630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9839798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9840182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9840374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9840394Z 
2023-01-11T22:07:08.9840502Z Running tests...
2023-01-11T22:07:08.9840768Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9841082Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9841344Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9841363Z 
2023-01-11T22:07:08.9841622Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9841736Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9841756Z 
2023-01-11T22:07:08.9841845Z OK (skipped=1)
2023-01-11T22:07:08.9841864Z 
2023-01-11T22:07:08.9841987Z Generating XML reports...
2023-01-11T22:07:08.9842431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214617.xml
2023-01-11T22:07:08.9842798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9842977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9843358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9843547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9843567Z 
2023-01-11T22:07:08.9843674Z Running tests...
2023-01-11T22:07:08.9843934Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9844233Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9844512Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T22:07:08.9844531Z 
2023-01-11T22:07:08.9844786Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9844896Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9844915Z 
2023-01-11T22:07:08.9845021Z OK (skipped=1)
2023-01-11T22:07:08.9845039Z 
2023-01-11T22:07:08.9845166Z Generating XML reports...
2023-01-11T22:07:08.9845612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214620.xml
2023-01-11T22:07:08.9845981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9846155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9846604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9846794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9846814Z 
2023-01-11T22:07:08.9846921Z Running tests...
2023-01-11T22:07:08.9847178Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9847487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9847829Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T22:07:08.9847851Z 
2023-01-11T22:07:08.9848116Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9848228Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9848248Z 
2023-01-11T22:07:08.9848337Z OK (skipped=1)
2023-01-11T22:07:08.9848373Z 
2023-01-11T22:07:08.9848524Z Generating XML reports...
2023-01-11T22:07:08.9848973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214622.xml
2023-01-11T22:07:08.9849341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9849513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9849886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9850079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9850099Z 
2023-01-11T22:07:08.9850207Z Running tests...
2023-01-11T22:07:08.9850467Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9850760Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9851031Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9851051Z 
2023-01-11T22:07:08.9851307Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9851416Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9851435Z 
2023-01-11T22:07:08.9851541Z OK (skipped=1)
2023-01-11T22:07:08.9851560Z 
2023-01-11T22:07:08.9851682Z Generating XML reports...
2023-01-11T22:07:08.9852124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214624.xml
2023-01-11T22:07:08.9852490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9852663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9853019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9853212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9853231Z 
2023-01-11T22:07:08.9853338Z Running tests...
2023-01-11T22:07:08.9853595Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9853902Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9854166Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9854389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25434
2023-01-11T22:07:08.9854605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25435
2023-01-11T22:07:08.9854959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9855133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9855509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9855761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9856125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9856294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9856664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9856891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9857147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9857375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9857780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9858178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9858407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9858634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9858735Z ok (6.950s)
2023-01-11T22:07:08.9858755Z 
2023-01-11T22:07:08.9859019Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9859131Z Ran 1 test in 6.950s
2023-01-11T22:07:08.9859151Z 
2023-01-11T22:07:08.9859226Z OK
2023-01-11T22:07:08.9859262Z 
2023-01-11T22:07:08.9859368Z Generating XML reports...
2023-01-11T22:07:08.9859817Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214627.xml
2023-01-11T22:07:08.9860189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9860367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9860745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9860935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9860955Z 
2023-01-11T22:07:08.9861063Z Running tests...
2023-01-11T22:07:08.9861324Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9861622Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9861881Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9861901Z 
2023-01-11T22:07:08.9862155Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9862270Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9862290Z 
2023-01-11T22:07:08.9862396Z OK (skipped=1)
2023-01-11T22:07:08.9862415Z 
2023-01-11T22:07:08.9862537Z Generating XML reports...
2023-01-11T22:07:08.9862982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214636.xml
2023-01-11T22:07:08.9863348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9863522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9863887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9864077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9864096Z 
2023-01-11T22:07:08.9864204Z Running tests...
2023-01-11T22:07:08.9864461Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9864830Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9865094Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9865114Z 
2023-01-11T22:07:08.9865371Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9865480Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9865500Z 
2023-01-11T22:07:08.9865607Z OK (skipped=1)
2023-01-11T22:07:08.9865626Z 
2023-01-11T22:07:08.9865732Z Generating XML reports...
2023-01-11T22:07:08.9866219Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214639.xml
2023-01-11T22:07:08.9866600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9866774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9867152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9867340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9867360Z 
2023-01-11T22:07:08.9867468Z Running tests...
2023-01-11T22:07:08.9867724Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9868019Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9868291Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:08.9868311Z 
2023-01-11T22:07:08.9868567Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9868677Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9868696Z 
2023-01-11T22:07:08.9868803Z OK (skipped=1)
2023-01-11T22:07:08.9868822Z 
2023-01-11T22:07:08.9868943Z Generating XML reports...
2023-01-11T22:07:08.9869394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214641.xml
2023-01-11T22:07:08.9869761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9869935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9870293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9870486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9870506Z 
2023-01-11T22:07:08.9870612Z Running tests...
2023-01-11T22:07:08.9870871Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9871183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9871444Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9871666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25648
2023-01-11T22:07:08.9871884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25649
2023-01-11T22:07:08.9872237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9872412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9872789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9872981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9873344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9873515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9873885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9874130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9874376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9874603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9875047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9875452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9875682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9875908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9876244Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9876568Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9876899Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9877226Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9877557Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9877898Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9878214Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9878526Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9878860Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9879202Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9879754Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9880115Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9880444Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9880750Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9881078Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9881416Z STAGE:2023-01-11 21:46:49 25649:25649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9881751Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9882088Z STAGE:2023-01-11 21:46:49 25648:25648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9882185Z ok (6.351s)
2023-01-11T22:07:08.9882206Z 
2023-01-11T22:07:08.9882462Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9882570Z Ran 1 test in 6.351s
2023-01-11T22:07:08.9882590Z 
2023-01-11T22:07:08.9882681Z OK
2023-01-11T22:07:08.9882705Z 
2023-01-11T22:07:08.9882814Z Generating XML reports...
2023-01-11T22:07:08.9883264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214643.xml
2023-01-11T22:07:08.9883627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9883800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9884267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9884456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9884476Z 
2023-01-11T22:07:08.9884581Z Running tests...
2023-01-11T22:07:08.9884839Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9885135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9885460Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9885689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25766
2023-01-11T22:07:08.9885900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25767
2023-01-11T22:07:08.9886268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9886443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9886821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9887007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9887369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9887528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9887900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9888086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9888325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9888568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9888959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9889345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9889565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9889778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9890106Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9890416Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9890741Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9891073Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9891411Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9891746Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9892062Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9892372Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9892685Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9893022Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9893346Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9893740Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9894055Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9894369Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9894688Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9895074Z STAGE:2023-01-11 21:46:58 25766:25766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9895405Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9895728Z STAGE:2023-01-11 21:46:58 25767:25767 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9895828Z ok (6.359s)
2023-01-11T22:07:08.9895848Z 
2023-01-11T22:07:08.9896101Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9896206Z Ran 1 test in 6.359s
2023-01-11T22:07:08.9896225Z 
2023-01-11T22:07:08.9896317Z OK
2023-01-11T22:07:08.9896337Z 
2023-01-11T22:07:08.9896456Z Generating XML reports...
2023-01-11T22:07:08.9896899Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214652.xml
2023-01-11T22:07:08.9897264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9897431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9897793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9897978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9897997Z 
2023-01-11T22:07:08.9898100Z Running tests...
2023-01-11T22:07:08.9898351Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9898652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9898920Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9899134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25884
2023-01-11T22:07:08.9899350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25885
2023-01-11T22:07:08.9899703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9899866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9900232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9900417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9900774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9900938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9901309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9901496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9901747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9901977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9902378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9902772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9903050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9903270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9903600Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9903921Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9904292Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9904637Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9904963Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9905310Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9905633Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9905948Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9906270Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9906605Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9906926Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9907261Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9907580Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9907894Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9908215Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9908550Z STAGE:2023-01-11 21:47:07 25884:25884 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9908874Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9909215Z STAGE:2023-01-11 21:47:07 25885:25885 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9909311Z ok (6.311s)
2023-01-11T22:07:08.9909331Z 
2023-01-11T22:07:08.9909591Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9909703Z Ran 1 test in 6.312s
2023-01-11T22:07:08.9909723Z 
2023-01-11T22:07:08.9909797Z OK
2023-01-11T22:07:08.9909833Z 
2023-01-11T22:07:08.9909940Z Generating XML reports...
2023-01-11T22:07:08.9910396Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214701.xml
2023-01-11T22:07:08.9910764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9910938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9911316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9911501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9911522Z 
2023-01-11T22:07:08.9911621Z Running tests...
2023-01-11T22:07:08.9911871Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9912167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9912400Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:07:08.9912471Z 
2023-01-11T22:07:08.9912728Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9912829Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9912848Z 
2023-01-11T22:07:08.9912944Z OK (skipped=1)
2023-01-11T22:07:08.9912963Z 
2023-01-11T22:07:08.9913078Z Generating XML reports...
2023-01-11T22:07:08.9913514Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214710.xml
2023-01-11T22:07:08.9913919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9914094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9914456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9914634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9914657Z 
2023-01-11T22:07:08.9914758Z Running tests...
2023-01-11T22:07:08.9915008Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9915310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9915558Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:07:08.9915578Z 
2023-01-11T22:07:08.9915826Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9915932Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9915951Z 
2023-01-11T22:07:08.9916041Z OK (skipped=1)
2023-01-11T22:07:08.9916073Z 
2023-01-11T22:07:08.9916178Z Generating XML reports...
2023-01-11T22:07:08.9916617Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214713.xml
2023-01-11T22:07:08.9916979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9917151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9917524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9917704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9917724Z 
2023-01-11T22:07:08.9917825Z Running tests...
2023-01-11T22:07:08.9918075Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9918375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9918622Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9918831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26068
2023-01-11T22:07:08.9919041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26069
2023-01-11T22:07:08.9919616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9919794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9920172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9920355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9920705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9920869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9921236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9921413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9921730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9921964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9922357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9922744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9923019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9923240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9923334Z ok (6.850s)
2023-01-11T22:07:08.9923355Z 
2023-01-11T22:07:08.9923616Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9923718Z Ran 1 test in 6.850s
2023-01-11T22:07:08.9923742Z 
2023-01-11T22:07:08.9923825Z OK
2023-01-11T22:07:08.9923844Z 
2023-01-11T22:07:08.9923958Z Generating XML reports...
2023-01-11T22:07:08.9924394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214715.xml
2023-01-11T22:07:08.9924757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9924922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9925286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9925469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9925489Z 
2023-01-11T22:07:08.9925590Z Running tests...
2023-01-11T22:07:08.9925837Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9926139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9926402Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9926613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26185
2023-01-11T22:07:08.9926819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26186
2023-01-11T22:07:08.9927171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9927340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9927707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9927885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9928239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9928408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9928771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9928947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9929188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9929417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9929808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9930196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9930416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9930692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9930782Z ok (6.925s)
2023-01-11T22:07:08.9930802Z 
2023-01-11T22:07:08.9931055Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9931158Z Ran 1 test in 6.925s
2023-01-11T22:07:08.9931177Z 
2023-01-11T22:07:08.9931251Z OK
2023-01-11T22:07:08.9931279Z 
2023-01-11T22:07:08.9931387Z Generating XML reports...
2023-01-11T22:07:08.9931871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214724.xml
2023-01-11T22:07:08.9932246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9932413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9932783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9932967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9932987Z 
2023-01-11T22:07:08.9933087Z Running tests...
2023-01-11T22:07:08.9933342Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9933637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9933886Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:07:08.9933905Z 
2023-01-11T22:07:08.9934163Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9934265Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9934285Z 
2023-01-11T22:07:08.9934390Z OK (skipped=1)
2023-01-11T22:07:08.9934410Z 
2023-01-11T22:07:08.9934523Z Generating XML reports...
2023-01-11T22:07:08.9934962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214734.xml
2023-01-11T22:07:08.9935327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9935491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9935855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9936034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9936053Z 
2023-01-11T22:07:08.9936154Z Running tests...
2023-01-11T22:07:08.9936407Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9936713Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9936971Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9937233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26335
2023-01-11T22:07:08.9937455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26336
2023-01-11T22:07:08.9937809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9937975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9938346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9938529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9938884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9939049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9939411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9939650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9939891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9940117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9940520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9940957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9941185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9941417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:08.9941641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9941876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:08.9942272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:08.9942656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:08.9942742Z ok (6.935s)
2023-01-11T22:07:08.9942763Z 
2023-01-11T22:07:08.9943025Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9943133Z Ran 1 test in 6.935s
2023-01-11T22:07:08.9943152Z 
2023-01-11T22:07:08.9943242Z OK
2023-01-11T22:07:08.9943261Z 
2023-01-11T22:07:08.9943382Z Generating XML reports...
2023-01-11T22:07:08.9943824Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214736.xml
2023-01-11T22:07:08.9944192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9944368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9944729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9944919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9944939Z 
2023-01-11T22:07:08.9945046Z Running tests...
2023-01-11T22:07:08.9945301Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9945604Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9945846Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:07:08.9945866Z 
2023-01-11T22:07:08.9946117Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9946224Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9946243Z 
2023-01-11T22:07:08.9946342Z OK (skipped=1)
2023-01-11T22:07:08.9946361Z 
2023-01-11T22:07:08.9946468Z Generating XML reports...
2023-01-11T22:07:08.9946905Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214746.xml
2023-01-11T22:07:08.9947266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9947430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9947803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9947985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9948004Z 
2023-01-11T22:07:08.9948105Z Running tests...
2023-01-11T22:07:08.9948367Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9948782Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9949029Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9949245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26487
2023-01-11T22:07:08.9949451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26488
2023-01-11T22:07:08.9949863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9950032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9950410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9950600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9950955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9951113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9951484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9951662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9951906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9952146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9952545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9952930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9953160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9953384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9953525Z skip: Skipped due to small world size. (4.236s)
2023-01-11T22:07:08.9953545Z 
2023-01-11T22:07:08.9953809Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9953909Z Ran 1 test in 4.237s
2023-01-11T22:07:08.9953928Z 
2023-01-11T22:07:08.9954033Z OK (skipped=1)
2023-01-11T22:07:08.9954052Z 
2023-01-11T22:07:08.9954177Z Generating XML reports...
2023-01-11T22:07:08.9954621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214748.xml
2023-01-11T22:07:08.9954988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9955156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9955536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9955709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9955730Z 
2023-01-11T22:07:08.9955829Z Running tests...
2023-01-11T22:07:08.9956086Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9956394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9956671Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:08.9956692Z 
2023-01-11T22:07:08.9956947Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9957051Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9957070Z 
2023-01-11T22:07:08.9957176Z OK (skipped=1)
2023-01-11T22:07:08.9957195Z 
2023-01-11T22:07:08.9957300Z Generating XML reports...
2023-01-11T22:07:08.9957802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214755.xml
2023-01-11T22:07:08.9958170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9958345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9958710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9958946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9958968Z 
2023-01-11T22:07:08.9959082Z Running tests...
2023-01-11T22:07:08.9959336Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9959990Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9960274Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:08.9960322Z 
2023-01-11T22:07:08.9960553Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9960662Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9960682Z 
2023-01-11T22:07:08.9960779Z OK (skipped=1)
2023-01-11T22:07:08.9960799Z 
2023-01-11T22:07:08.9960926Z Generating XML reports...
2023-01-11T22:07:08.9961363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214757.xml
2023-01-11T22:07:08.9961728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9961900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9962271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9962445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9962468Z 
2023-01-11T22:07:08.9962574Z Running tests...
2023-01-11T22:07:08.9962825Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9963134Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9963410Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9964161Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2023-01-11T22:07:08.9964183Z 
2023-01-11T22:07:08.9964433Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9964545Z Ran 1 test in 1.646s
2023-01-11T22:07:08.9964567Z 
2023-01-11T22:07:08.9964673Z OK (skipped=1)
2023-01-11T22:07:08.9964692Z 
2023-01-11T22:07:08.9964806Z Generating XML reports...
2023-01-11T22:07:08.9965241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214800.xml
2023-01-11T22:07:08.9965607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9965770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9966149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9966332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9966352Z 
2023-01-11T22:07:08.9966458Z Running tests...
2023-01-11T22:07:08.9966711Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9967020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9967387Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9967602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26692
2023-01-11T22:07:08.9967816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26693
2023-01-11T22:07:08.9968188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9968408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9968797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9968978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9969337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9969505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9969860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9970050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9970292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9970537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9970935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9971321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9971549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9971775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9972113Z STAGE:2023-01-11 21:48:09 26692:26692 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9972417Z STAGE:2023-01-11 21:48:10 26693:26693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:08.9972746Z STAGE:2023-01-11 21:48:11 26692:26692 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9973072Z STAGE:2023-01-11 21:48:11 26693:26693 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:08.9973414Z STAGE:2023-01-11 21:48:11 26693:26693 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9973753Z STAGE:2023-01-11 21:48:11 26692:26692 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:08.9973855Z ok (7.828s)
2023-01-11T22:07:08.9973880Z 
2023-01-11T22:07:08.9974135Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9974245Z Ran 1 test in 7.828s
2023-01-11T22:07:08.9974264Z 
2023-01-11T22:07:08.9974339Z OK
2023-01-11T22:07:08.9974373Z 
2023-01-11T22:07:08.9974481Z Generating XML reports...
2023-01-11T22:07:08.9974922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214804.xml
2023-01-11T22:07:08.9975291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9975457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9975833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9976017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9976037Z 
2023-01-11T22:07:08.9976144Z Running tests...
2023-01-11T22:07:08.9976465Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9976762Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9977054Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:08.9977074Z 
2023-01-11T22:07:08.9977330Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9977430Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9977449Z 
2023-01-11T22:07:08.9977596Z OK (skipped=1)
2023-01-11T22:07:08.9977617Z 
2023-01-11T22:07:08.9977742Z Generating XML reports...
2023-01-11T22:07:08.9978182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214814.xml
2023-01-11T22:07:08.9978549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9978720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9979085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9979275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9979295Z 
2023-01-11T22:07:08.9979400Z Running tests...
2023-01-11T22:07:08.9979652Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9979964Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9980249Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9980999Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2023-01-11T22:07:08.9981023Z 
2023-01-11T22:07:08.9981281Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9981386Z Ran 1 test in 1.625s
2023-01-11T22:07:08.9981406Z 
2023-01-11T22:07:08.9981497Z OK (skipped=1)
2023-01-11T22:07:08.9981533Z 
2023-01-11T22:07:08.9981639Z Generating XML reports...
2023-01-11T22:07:08.9982079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214817.xml
2023-01-11T22:07:08.9982449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9982614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9982989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9983174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9983193Z 
2023-01-11T22:07:08.9983301Z Running tests...
2023-01-11T22:07:08.9983561Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9983856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9984143Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:08.9984164Z 
2023-01-11T22:07:08.9984423Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9984525Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9984544Z 
2023-01-11T22:07:08.9984650Z OK (skipped=1)
2023-01-11T22:07:08.9984668Z 
2023-01-11T22:07:08.9984785Z Generating XML reports...
2023-01-11T22:07:08.9985236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214821.xml
2023-01-11T22:07:08.9985660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9985835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9986198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9986385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9986406Z 
2023-01-11T22:07:08.9986504Z Running tests...
2023-01-11T22:07:08.9986805Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9987116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9987403Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:08.9987614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26913
2023-01-11T22:07:08.9987833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26914
2023-01-11T22:07:08.9988183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9988351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9988726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9988912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9989276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9989447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9989813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9990005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9990241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:08.9990467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:08.9990866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9991257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:08.9991488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:08.9991706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:08.9991866Z skip: Skipped due to small world size. (4.243s)
2023-01-11T22:07:08.9991886Z 
2023-01-11T22:07:08.9992145Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9992255Z Ran 1 test in 4.243s
2023-01-11T22:07:08.9992275Z 
2023-01-11T22:07:08.9992380Z OK (skipped=1)
2023-01-11T22:07:08.9992399Z 
2023-01-11T22:07:08.9992505Z Generating XML reports...
2023-01-11T22:07:08.9992949Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214823.xml
2023-01-11T22:07:08.9993320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9993490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9993867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9994055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9994075Z 
2023-01-11T22:07:08.9994175Z Running tests...
2023-01-11T22:07:08.9994490Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9994783Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9995060Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:08.9995081Z 
2023-01-11T22:07:08.9995334Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9995436Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9995455Z 
2023-01-11T22:07:08.9995606Z OK (skipped=1)
2023-01-11T22:07:08.9995627Z 
2023-01-11T22:07:08.9995747Z Generating XML reports...
2023-01-11T22:07:08.9996196Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214830.xml
2023-01-11T22:07:08.9996554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:08.9996729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:08.9997087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:08.9997267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:08.9997287Z 
2023-01-11T22:07:08.9997394Z Running tests...
2023-01-11T22:07:08.9997645Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9997955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:08.9998253Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:08.9998273Z 
2023-01-11T22:07:08.9998528Z ----------------------------------------------------------------------
2023-01-11T22:07:08.9998638Z Ran 1 test in 0.002s
2023-01-11T22:07:08.9998657Z 
2023-01-11T22:07:08.9998766Z OK (skipped=1)
2023-01-11T22:07:08.9998785Z 
2023-01-11T22:07:08.9998890Z Generating XML reports...
2023-01-11T22:07:08.9999337Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214832.xml
2023-01-11T22:07:08.9999954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0000126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0000508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0000697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0000717Z 
2023-01-11T22:07:09.0000821Z Running tests...
2023-01-11T22:07:09.0001081Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0001391Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0001663Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0001881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27084
2023-01-11T22:07:09.0002096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27085
2023-01-11T22:07:09.0002464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0002641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0003018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0003204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0003561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0003799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0004176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0004363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0004608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0004906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0005315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0005710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0005936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0006166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0006252Z ok (6.936s)
2023-01-11T22:07:09.0006272Z 
2023-01-11T22:07:09.0006538Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0006648Z Ran 1 test in 6.936s
2023-01-11T22:07:09.0006668Z 
2023-01-11T22:07:09.0006761Z OK
2023-01-11T22:07:09.0006781Z 
2023-01-11T22:07:09.0006904Z Generating XML reports...
2023-01-11T22:07:09.0007355Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214835.xml
2023-01-11T22:07:09.0007727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0007901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0008279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0008457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0008477Z 
2023-01-11T22:07:09.0008583Z Running tests...
2023-01-11T22:07:09.0008843Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0009153Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0009446Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0009667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27201
2023-01-11T22:07:09.0009884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27202
2023-01-11T22:07:09.0010251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0010409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0010791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0010978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0011340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0011508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0011880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0012063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0012309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0012553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0012998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0013392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0013620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0013847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0013948Z ok (6.928s)
2023-01-11T22:07:09.0013968Z 
2023-01-11T22:07:09.0014271Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0014390Z Ran 1 test in 6.928s
2023-01-11T22:07:09.0014410Z 
2023-01-11T22:07:09.0014501Z OK
2023-01-11T22:07:09.0014520Z 
2023-01-11T22:07:09.0014626Z Generating XML reports...
2023-01-11T22:07:09.0015077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214844.xml
2023-01-11T22:07:09.0015448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0015623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0015996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0016184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0016203Z 
2023-01-11T22:07:09.0016310Z Running tests...
2023-01-11T22:07:09.0016570Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0016881Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0017165Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:09.0017201Z 
2023-01-11T22:07:09.0017442Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0017551Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0017571Z 
2023-01-11T22:07:09.0017677Z OK (skipped=1)
2023-01-11T22:07:09.0017696Z 
2023-01-11T22:07:09.0017817Z Generating XML reports...
2023-01-11T22:07:09.0018263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214854.xml
2023-01-11T22:07:09.0018633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0018808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0019185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0019360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0019379Z 
2023-01-11T22:07:09.0019487Z Running tests...
2023-01-11T22:07:09.0019751Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0020064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0020357Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0020575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27351
2023-01-11T22:07:09.0020793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27352
2023-01-11T22:07:09.0021166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0021323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0021703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0021950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0022317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0022487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0022854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0023041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0023338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0023587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0023973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0024370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0024602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0024842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0025066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0025300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0025699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0026091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0026194Z ok (7.024s)
2023-01-11T22:07:09.0026213Z 
2023-01-11T22:07:09.0026457Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0026571Z Ran 1 test in 7.024s
2023-01-11T22:07:09.0026590Z 
2023-01-11T22:07:09.0026680Z OK
2023-01-11T22:07:09.0026699Z 
2023-01-11T22:07:09.0026819Z Generating XML reports...
2023-01-11T22:07:09.0027263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214856.xml
2023-01-11T22:07:09.0027631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0027806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0028182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0028370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0028389Z 
2023-01-11T22:07:09.0028480Z Running tests...
2023-01-11T22:07:09.0028737Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0029053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0029345Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:07:09.0029365Z 
2023-01-11T22:07:09.0029625Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0029734Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0029754Z 
2023-01-11T22:07:09.0029860Z OK (skipped=1)
2023-01-11T22:07:09.0029883Z 
2023-01-11T22:07:09.0030006Z Generating XML reports...
2023-01-11T22:07:09.0030436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214906.xml
2023-01-11T22:07:09.0030805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0030976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0031415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0031605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0031625Z 
2023-01-11T22:07:09.0031731Z Running tests...
2023-01-11T22:07:09.0031992Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0032301Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0032628Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0032838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27503
2023-01-11T22:07:09.0033053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27504
2023-01-11T22:07:09.0033421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0033599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0033975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0034163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0034523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0034697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0035073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0035244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0035489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0035733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0036131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0036521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0036749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0036978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0037136Z skip: Skipped due to small world size. (4.242s)
2023-01-11T22:07:09.0037155Z 
2023-01-11T22:07:09.0037401Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0037515Z Ran 1 test in 4.242s
2023-01-11T22:07:09.0037534Z 
2023-01-11T22:07:09.0037639Z OK (skipped=1)
2023-01-11T22:07:09.0037661Z 
2023-01-11T22:07:09.0037783Z Generating XML reports...
2023-01-11T22:07:09.0038229Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214908.xml
2023-01-11T22:07:09.0038595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0038767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0039147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0039336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0039355Z 
2023-01-11T22:07:09.0039689Z Running tests...
2023-01-11T22:07:09.0039950Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0040262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0040604Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0040820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27608
2023-01-11T22:07:09.0041035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27609
2023-01-11T22:07:09.0041405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0041577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0041992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0042193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0042557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0042730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0043108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0043292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0043536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0043777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0044179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0044561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0044790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0045015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0045258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0045498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0045897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0046289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0046391Z ok (6.850s)
2023-01-11T22:07:09.0046411Z 
2023-01-11T22:07:09.0046675Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0046769Z Ran 1 test in 6.850s
2023-01-11T22:07:09.0046788Z 
2023-01-11T22:07:09.0046879Z OK
2023-01-11T22:07:09.0046899Z 
2023-01-11T22:07:09.0047021Z Generating XML reports...
2023-01-11T22:07:09.0047476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214915.xml
2023-01-11T22:07:09.0047843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0048015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0048394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0048629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0048651Z 
2023-01-11T22:07:09.0048760Z Running tests...
2023-01-11T22:07:09.0049002Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0049313Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0049573Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0049848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27732
2023-01-11T22:07:09.0050063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27733
2023-01-11T22:07:09.0050432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0050607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0051028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0051207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0051573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0051744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0052115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0052304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0052550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0052793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0053197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0053593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0053804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0054031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0054183Z skip: Need at least 3 CUDA devices (4.228s)
2023-01-11T22:07:09.0054203Z 
2023-01-11T22:07:09.0054467Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0054577Z Ran 1 test in 4.229s
2023-01-11T22:07:09.0054596Z 
2023-01-11T22:07:09.0054702Z OK (skipped=1)
2023-01-11T22:07:09.0054721Z 
2023-01-11T22:07:09.0054844Z Generating XML reports...
2023-01-11T22:07:09.0055294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214924.xml
2023-01-11T22:07:09.0055649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0055822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0056200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0056390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0056413Z 
2023-01-11T22:07:09.0056519Z Running tests...
2023-01-11T22:07:09.0056780Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0057090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0057339Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2023-01-11T22:07:09.0057359Z 
2023-01-11T22:07:09.0057619Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0057717Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0057736Z 
2023-01-11T22:07:09.0057844Z OK (skipped=1)
2023-01-11T22:07:09.0057863Z 
2023-01-11T22:07:09.0057983Z Generating XML reports...
2023-01-11T22:07:09.0058432Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214931.xml
2023-01-11T22:07:09.0058800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0059029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0059408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0059596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0059616Z 
2023-01-11T22:07:09.0059724Z Running tests...
2023-01-11T22:07:09.0059968Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0060323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0060582Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T22:07:09.0060603Z 
2023-01-11T22:07:09.0060863Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0060973Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0060996Z 
2023-01-11T22:07:09.0061102Z OK (skipped=1)
2023-01-11T22:07:09.0061121Z 
2023-01-11T22:07:09.0061243Z Generating XML reports...
2023-01-11T22:07:09.0061688Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214933.xml
2023-01-11T22:07:09.0062059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0062218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0062598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0062788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0062808Z 
2023-01-11T22:07:09.0062916Z Running tests...
2023-01-11T22:07:09.0063179Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0063490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0063746Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0063966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27903
2023-01-11T22:07:09.0064164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27904
2023-01-11T22:07:09.0064535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0064710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0065087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0065275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0065635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0065810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0066179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0066348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0066593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0066839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0067240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0067632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0067863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0068150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0068252Z ok (7.742s)
2023-01-11T22:07:09.0068272Z 
2023-01-11T22:07:09.0068542Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0068635Z Ran 1 test in 7.742s
2023-01-11T22:07:09.0068654Z 
2023-01-11T22:07:09.0068743Z OK
2023-01-11T22:07:09.0068762Z 
2023-01-11T22:07:09.0068881Z Generating XML reports...
2023-01-11T22:07:09.0069375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214936.xml
2023-01-11T22:07:09.0069754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0069928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0070306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0070498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0070518Z 
2023-01-11T22:07:09.0070626Z Running tests...
2023-01-11T22:07:09.0070869Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0071181Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0071446Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T22:07:09.0071470Z 
2023-01-11T22:07:09.0071729Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0071839Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0071859Z 
2023-01-11T22:07:09.0071967Z OK (skipped=1)
2023-01-11T22:07:09.0071986Z 
2023-01-11T22:07:09.0072108Z Generating XML reports...
2023-01-11T22:07:09.0072553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214946.xml
2023-01-11T22:07:09.0072926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0073082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0073457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0073644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0073663Z 
2023-01-11T22:07:09.0073772Z Running tests...
2023-01-11T22:07:09.0074029Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0074334Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0074600Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0074819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28051
2023-01-11T22:07:09.0075022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28052
2023-01-11T22:07:09.0075387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0075558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0075934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0076122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0076479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0076649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0077019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0077262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0077491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0077730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0078131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0078600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0078838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0079065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0079223Z skip: Skipped due to small world size. (4.221s)
2023-01-11T22:07:09.0079246Z 
2023-01-11T22:07:09.0079732Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0079833Z Ran 1 test in 4.221s
2023-01-11T22:07:09.0079870Z 
2023-01-11T22:07:09.0079962Z OK (skipped=1)
2023-01-11T22:07:09.0079981Z 
2023-01-11T22:07:09.0080104Z Generating XML reports...
2023-01-11T22:07:09.0080561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214949.xml
2023-01-11T22:07:09.0080933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0081109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0081489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0081678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0081697Z 
2023-01-11T22:07:09.0081810Z Running tests...
2023-01-11T22:07:09.0082054Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0082363Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0082619Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T22:07:09.0082639Z 
2023-01-11T22:07:09.0082894Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0083004Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0083023Z 
2023-01-11T22:07:09.0083199Z OK (skipped=1)
2023-01-11T22:07:09.0083219Z 
2023-01-11T22:07:09.0083342Z Generating XML reports...
2023-01-11T22:07:09.0083787Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214955.xml
2023-01-11T22:07:09.0084157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0084319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0084699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0084888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0084907Z 
2023-01-11T22:07:09.0085013Z Running tests...
2023-01-11T22:07:09.0085273Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0085585Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0085843Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0086060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28189
2023-01-11T22:07:09.0086262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28190
2023-01-11T22:07:09.0086636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0086893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0087274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0087463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0087823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0088060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0088448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0088635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0088865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0089112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0089510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0089903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0090131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0090360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0090517Z skip: Skipped due to small world size. (4.149s)
2023-01-11T22:07:09.0090538Z 
2023-01-11T22:07:09.0090802Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0090910Z Ran 1 test in 4.150s
2023-01-11T22:07:09.0090930Z 
2023-01-11T22:07:09.0091025Z OK (skipped=1)
2023-01-11T22:07:09.0091045Z 
2023-01-11T22:07:09.0091167Z Generating XML reports...
2023-01-11T22:07:09.0091616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214958.xml
2023-01-11T22:07:09.0091986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0092160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0092539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0092728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0092748Z 
2023-01-11T22:07:09.0092855Z Running tests...
2023-01-11T22:07:09.0093100Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0093414Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0093696Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T22:07:09.0093715Z 
2023-01-11T22:07:09.0093970Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0094081Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0094100Z 
2023-01-11T22:07:09.0094208Z OK (skipped=1)
2023-01-11T22:07:09.0094227Z 
2023-01-11T22:07:09.0094349Z Generating XML reports...
2023-01-11T22:07:09.0094798Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215004.xml
2023-01-11T22:07:09.0095170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0095328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0095703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0095951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0095971Z 
2023-01-11T22:07:09.0096078Z Running tests...
2023-01-11T22:07:09.0096338Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0096648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0096921Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T22:07:09.0096984Z 
2023-01-11T22:07:09.0097250Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0097361Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0097380Z 
2023-01-11T22:07:09.0097471Z OK (skipped=1)
2023-01-11T22:07:09.0097490Z 
2023-01-11T22:07:09.0097613Z Generating XML reports...
2023-01-11T22:07:09.0098062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215007.xml
2023-01-11T22:07:09.0098436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0098610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0098983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0099170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0099190Z 
2023-01-11T22:07:09.0099301Z Running tests...
2023-01-11T22:07:09.0099544Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0099856Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0100130Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T22:07:09.0100149Z 
2023-01-11T22:07:09.0100410Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0100521Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0100540Z 
2023-01-11T22:07:09.0100645Z OK (skipped=1)
2023-01-11T22:07:09.0100665Z 
2023-01-11T22:07:09.0100786Z Generating XML reports...
2023-01-11T22:07:09.0101232Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215009.xml
2023-01-11T22:07:09.0101603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0101759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0102137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0102328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0102347Z 
2023-01-11T22:07:09.0102455Z Running tests...
2023-01-11T22:07:09.0102719Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0103029Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0103284Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T22:07:09.0103304Z 
2023-01-11T22:07:09.0103556Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0103667Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0103686Z 
2023-01-11T22:07:09.0103780Z OK (skipped=1)
2023-01-11T22:07:09.0103799Z 
2023-01-11T22:07:09.0103921Z Generating XML reports...
2023-01-11T22:07:09.0104373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215012.xml
2023-01-11T22:07:09.0104739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0104972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0105350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0105538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0105558Z 
2023-01-11T22:07:09.0105665Z Running tests...
2023-01-11T22:07:09.0105923Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0106261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0106534Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T22:07:09.0106554Z 
2023-01-11T22:07:09.0106810Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0106920Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0106940Z 
2023-01-11T22:07:09.0107047Z OK (skipped=1)
2023-01-11T22:07:09.0107070Z 
2023-01-11T22:07:09.0107192Z Generating XML reports...
2023-01-11T22:07:09.0107636Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215014.xml
2023-01-11T22:07:09.0108001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0108158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0108537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0108726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0108745Z 
2023-01-11T22:07:09.0108852Z Running tests...
2023-01-11T22:07:09.0109111Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0109421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0109704Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0109923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28459
2023-01-11T22:07:09.0110139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28460
2023-01-11T22:07:09.0110492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0110665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0111045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0111232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0111594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0111769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0112142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0112324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0112556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0112798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0113198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0113593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0113822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0114109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0114346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0114586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0114986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0115426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0115657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:09.0115893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:09.0116284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0116677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0116780Z ok (4.343s)
2023-01-11T22:07:09.0116800Z 
2023-01-11T22:07:09.0117060Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0117169Z Ran 1 test in 4.343s
2023-01-11T22:07:09.0117189Z 
2023-01-11T22:07:09.0117282Z OK
2023-01-11T22:07:09.0117301Z 
2023-01-11T22:07:09.0117407Z Generating XML reports...
2023-01-11T22:07:09.0117862Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215016.xml
2023-01-11T22:07:09.0118228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0118402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0118780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0118974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0118994Z 
2023-01-11T22:07:09.0119102Z Running tests...
2023-01-11T22:07:09.0119361Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0119902Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0120157Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0120378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28572
2023-01-11T22:07:09.0120593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28573
2023-01-11T22:07:09.0120963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0121134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0121514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0121702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0122062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0122216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0122590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0122779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0123024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0123268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0123753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0124148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0124374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0124601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0124687Z ok (5.855s)
2023-01-11T22:07:09.0124762Z 
2023-01-11T22:07:09.0125037Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0125149Z Ran 1 test in 5.855s
2023-01-11T22:07:09.0125168Z 
2023-01-11T22:07:09.0125259Z OK
2023-01-11T22:07:09.0125279Z 
2023-01-11T22:07:09.0125402Z Generating XML reports...
2023-01-11T22:07:09.0125851Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215023.xml
2023-01-11T22:07:09.0126228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0126403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0126779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0126953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0126972Z 
2023-01-11T22:07:09.0127086Z Running tests...
2023-01-11T22:07:09.0127343Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0127653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0127930Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0128152Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28688
2023-01-11T22:07:09.0128367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28689
2023-01-11T22:07:09.0128736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0128891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0129269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0129460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0129822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0129993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0130362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0130551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0130795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0131037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0131421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0131820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0132046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0132273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0132429Z skip: Skipped due to small world size. (4.228s)
2023-01-11T22:07:09.0132504Z 
2023-01-11T22:07:09.0132772Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0132884Z Ran 1 test in 4.228s
2023-01-11T22:07:09.0132903Z 
2023-01-11T22:07:09.0133011Z OK (skipped=1)
2023-01-11T22:07:09.0133030Z 
2023-01-11T22:07:09.0133153Z Generating XML reports...
2023-01-11T22:07:09.0133586Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215032.xml
2023-01-11T22:07:09.0133996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0134177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0134557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0134746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0134766Z 
2023-01-11T22:07:09.0134878Z Running tests...
2023-01-11T22:07:09.0135139Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0135452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0135702Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0135920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28793
2023-01-11T22:07:09.0136137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28794
2023-01-11T22:07:09.0136506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0136678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0137054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0137247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0137614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0137786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0138142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0138328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0138574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0138815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0139212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0139608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0139840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0140066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0140150Z ok (5.552s)
2023-01-11T22:07:09.0140188Z 
2023-01-11T22:07:09.0140435Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0140546Z Ran 1 test in 5.552s
2023-01-11T22:07:09.0140566Z 
2023-01-11T22:07:09.0140661Z OK
2023-01-11T22:07:09.0140681Z 
2023-01-11T22:07:09.0140804Z Generating XML reports...
2023-01-11T22:07:09.0141254Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215038.xml
2023-01-11T22:07:09.0141623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0141855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0142235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0142409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0142428Z 
2023-01-11T22:07:09.0142539Z Running tests...
2023-01-11T22:07:09.0142795Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0143147Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0143427Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0143645Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28899
2023-01-11T22:07:09.0143860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28900
2023-01-11T22:07:09.0144234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0144390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0144769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0144959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0145322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0145494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0145866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0146051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0146294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0146541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0146925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0147317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0147549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0147778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0147879Z ok (4.432s)
2023-01-11T22:07:09.0147899Z 
2023-01-11T22:07:09.0148166Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0148277Z Ran 1 test in 4.432s
2023-01-11T22:07:09.0148296Z 
2023-01-11T22:07:09.0148388Z OK
2023-01-11T22:07:09.0148410Z 
2023-01-11T22:07:09.0148575Z Generating XML reports...
2023-01-11T22:07:09.0149014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215047.xml
2023-01-11T22:07:09.0149381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0149555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0149935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0150123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0150142Z 
2023-01-11T22:07:09.0150250Z Running tests...
2023-01-11T22:07:09.0150509Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0150819Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0151145Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0151363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29004
2023-01-11T22:07:09.0151576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29005
2023-01-11T22:07:09.0151948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0152121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0152555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0152754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0153120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0153297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0153654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0153840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0154086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0154329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0154731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0155126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0155353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0155582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0155667Z ok (5.954s)
2023-01-11T22:07:09.0155703Z 
2023-01-11T22:07:09.0155951Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0156060Z Ran 1 test in 5.954s
2023-01-11T22:07:09.0156079Z 
2023-01-11T22:07:09.0156171Z OK
2023-01-11T22:07:09.0156190Z 
2023-01-11T22:07:09.0156315Z Generating XML reports...
2023-01-11T22:07:09.0156767Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215054.xml
2023-01-11T22:07:09.0157136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0157311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0157689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0157868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0157887Z 
2023-01-11T22:07:09.0157998Z Running tests...
2023-01-11T22:07:09.0158258Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0158568Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0158838Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0159058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29120
2023-01-11T22:07:09.0159272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29121
2023-01-11T22:07:09.0159874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0160034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0160416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0160687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0161054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0161224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0161593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0161835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0162088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0162334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0162720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0163120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0163342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0163567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0163669Z ok (7.066s)
2023-01-11T22:07:09.0163689Z 
2023-01-11T22:07:09.0163954Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0164064Z Ran 1 test in 7.067s
2023-01-11T22:07:09.0164083Z 
2023-01-11T22:07:09.0164174Z OK
2023-01-11T22:07:09.0164194Z 
2023-01-11T22:07:09.0164317Z Generating XML reports...
2023-01-11T22:07:09.0164752Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215102.xml
2023-01-11T22:07:09.0165129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0165302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0165681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0165870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0165889Z 
2023-01-11T22:07:09.0165998Z Running tests...
2023-01-11T22:07:09.0166262Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0166577Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0166831Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0167050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29235
2023-01-11T22:07:09.0167269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29236
2023-01-11T22:07:09.0167642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0167815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0168188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0168379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0168740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0168913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0169268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0169455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0169756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0170000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0170401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0170793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0171066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0171300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0171385Z ok (4.327s)
2023-01-11T22:07:09.0171421Z 
2023-01-11T22:07:09.0171671Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0171786Z Ran 1 test in 4.327s
2023-01-11T22:07:09.0171807Z 
2023-01-11T22:07:09.0171900Z OK
2023-01-11T22:07:09.0171919Z 
2023-01-11T22:07:09.0172042Z Generating XML reports...
2023-01-11T22:07:09.0172491Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215112.xml
2023-01-11T22:07:09.0172860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0173033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0173414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0173589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0173609Z 
2023-01-11T22:07:09.0173717Z Running tests...
2023-01-11T22:07:09.0173976Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0174293Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0174546Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0174565Z 
2023-01-11T22:07:09.0174820Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0174930Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0174950Z 
2023-01-11T22:07:09.0175058Z OK (skipped=1)
2023-01-11T22:07:09.0175077Z 
2023-01-11T22:07:09.0175200Z Generating XML reports...
2023-01-11T22:07:09.0175635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215119.xml
2023-01-11T22:07:09.0176004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0176177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0176555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0176748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0176767Z 
2023-01-11T22:07:09.0176874Z Running tests...
2023-01-11T22:07:09.0177129Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0177439Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0177680Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0178428Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T22:07:09.0178467Z 
2023-01-11T22:07:09.0178771Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0178882Z Ran 1 test in 1.633s
2023-01-11T22:07:09.0178902Z 
2023-01-11T22:07:09.0179009Z OK (skipped=1)
2023-01-11T22:07:09.0179028Z 
2023-01-11T22:07:09.0179149Z Generating XML reports...
2023-01-11T22:07:09.0179594Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215121.xml
2023-01-11T22:07:09.0179962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0180180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0180569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0180744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0180764Z 
2023-01-11T22:07:09.0180871Z Running tests...
2023-01-11T22:07:09.0181137Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0181452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0181715Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0181736Z 
2023-01-11T22:07:09.0181993Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0182105Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0182124Z 
2023-01-11T22:07:09.0182232Z OK (skipped=1)
2023-01-11T22:07:09.0182254Z 
2023-01-11T22:07:09.0182376Z Generating XML reports...
2023-01-11T22:07:09.0182804Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215125.xml
2023-01-11T22:07:09.0183166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0183338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0183719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0183908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0183928Z 
2023-01-11T22:07:09.0184036Z Running tests...
2023-01-11T22:07:09.0184295Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0184607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0184854Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0184890Z 
2023-01-11T22:07:09.0185130Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0185240Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0185259Z 
2023-01-11T22:07:09.0185365Z OK (skipped=1)
2023-01-11T22:07:09.0185384Z 
2023-01-11T22:07:09.0185511Z Generating XML reports...
2023-01-11T22:07:09.0185957Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215127.xml
2023-01-11T22:07:09.0186324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0186497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0186875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0187052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0187073Z 
2023-01-11T22:07:09.0187179Z Running tests...
2023-01-11T22:07:09.0187440Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0187752Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0188075Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2023-01-11T22:07:09.0188095Z 
2023-01-11T22:07:09.0188351Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0188461Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0188481Z 
2023-01-11T22:07:09.0188585Z OK (skipped=1)
2023-01-11T22:07:09.0188605Z 
2023-01-11T22:07:09.0188731Z Generating XML reports...
2023-01-11T22:07:09.0189160Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215130.xml
2023-01-11T22:07:09.0189571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0189752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0190131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0190323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0190343Z 
2023-01-11T22:07:09.0190451Z Running tests...
2023-01-11T22:07:09.0190711Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0191020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0191267Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0192019Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2023-01-11T22:07:09.0192057Z 
2023-01-11T22:07:09.0192301Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0192412Z Ran 1 test in 1.636s
2023-01-11T22:07:09.0192435Z 
2023-01-11T22:07:09.0192542Z OK (skipped=1)
2023-01-11T22:07:09.0192561Z 
2023-01-11T22:07:09.0192682Z Generating XML reports...
2023-01-11T22:07:09.0193125Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215132.xml
2023-01-11T22:07:09.0193491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0193666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0194045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0194217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0194236Z 
2023-01-11T22:07:09.0194343Z Running tests...
2023-01-11T22:07:09.0194605Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0194915Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0195232Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0195970Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.639s)
2023-01-11T22:07:09.0195991Z 
2023-01-11T22:07:09.0196255Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0196366Z Ran 1 test in 1.639s
2023-01-11T22:07:09.0196385Z 
2023-01-11T22:07:09.0196491Z OK (skipped=1)
2023-01-11T22:07:09.0196510Z 
2023-01-11T22:07:09.0196632Z Generating XML reports...
2023-01-11T22:07:09.0197061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215136.xml
2023-01-11T22:07:09.0197487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0197664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0198042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0198231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0198250Z 
2023-01-11T22:07:09.0198398Z Running tests...
2023-01-11T22:07:09.0198666Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0198978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0199278Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0200266Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2023-01-11T22:07:09.0200308Z 
2023-01-11T22:07:09.0200550Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0200662Z Ran 1 test in 1.635s
2023-01-11T22:07:09.0200681Z 
2023-01-11T22:07:09.0200791Z OK (skipped=1)
2023-01-11T22:07:09.0200809Z 
2023-01-11T22:07:09.0200936Z Generating XML reports...
2023-01-11T22:07:09.0201383Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215141.xml
2023-01-11T22:07:09.0201749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0201922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0202305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0202477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0202514Z 
2023-01-11T22:07:09.0202605Z Running tests...
2023-01-11T22:07:09.0202861Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0203171Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0203448Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0203667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29608
2023-01-11T22:07:09.0203884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29609
2023-01-11T22:07:09.0204256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0204432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0204795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0204984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0205351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0205527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0205898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0206083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0206328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0206656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0207042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0207438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0207669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0207958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0208754Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:07:09.0208872Z   warnings.warn(
2023-01-11T22:07:09.0209652Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:07:09.0209764Z   warnings.warn(
2023-01-11T22:07:09.0210000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0210231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0210451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0210685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0210914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0211143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0211366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0211593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0211695Z ok (7.832s)
2023-01-11T22:07:09.0211715Z 
2023-01-11T22:07:09.0211982Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0212077Z Ran 1 test in 7.832s
2023-01-11T22:07:09.0212117Z 
2023-01-11T22:07:09.0212194Z OK
2023-01-11T22:07:09.0212212Z 
2023-01-11T22:07:09.0212337Z Generating XML reports...
2023-01-11T22:07:09.0212787Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215145.xml
2023-01-11T22:07:09.0213156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0213335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0213713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0213906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0213925Z 
2023-01-11T22:07:09.0214035Z Running tests...
2023-01-11T22:07:09.0214277Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0214593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0214900Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0215119Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29726
2023-01-11T22:07:09.0215334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29727
2023-01-11T22:07:09.0215767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0215943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0216323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0216514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0216921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0217102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0217479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0217662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0217907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0218152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0218552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0218946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0219179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0219391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0220170Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:07:09.0220288Z   warnings.warn(
2023-01-11T22:07:09.0221061Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:07:09.0221171Z   warnings.warn(
2023-01-11T22:07:09.0221409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0221636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0221869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0222101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0222189Z ok (6.865s)
2023-01-11T22:07:09.0222225Z 
2023-01-11T22:07:09.0222475Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0222586Z Ran 1 test in 6.866s
2023-01-11T22:07:09.0222606Z 
2023-01-11T22:07:09.0222698Z OK
2023-01-11T22:07:09.0222717Z 
2023-01-11T22:07:09.0222841Z Generating XML reports...
2023-01-11T22:07:09.0223293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215155.xml
2023-01-11T22:07:09.0223662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0223839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0224219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0224392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0224499Z 
2023-01-11T22:07:09.0224612Z Running tests...
2023-01-11T22:07:09.0224875Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0225185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0225480Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0225698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29844
2023-01-11T22:07:09.0225954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29845
2023-01-11T22:07:09.0226330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0226488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0226863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0227056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0227426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0227596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0227971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0228158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0228405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0228650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0229042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0229445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0229664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0229891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0230666Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:07:09.0230779Z   warnings.warn(
2023-01-11T22:07:09.0231552Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:07:09.0231664Z   warnings.warn(
2023-01-11T22:07:09.0231764Z ok (7.526s)
2023-01-11T22:07:09.0231784Z 
2023-01-11T22:07:09.0232047Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0232142Z Ran 1 test in 7.527s
2023-01-11T22:07:09.0232162Z 
2023-01-11T22:07:09.0232253Z OK
2023-01-11T22:07:09.0232272Z 
2023-01-11T22:07:09.0232394Z Generating XML reports...
2023-01-11T22:07:09.0232846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215205.xml
2023-01-11T22:07:09.0233216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0233391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0233767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0234019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0234039Z 
2023-01-11T22:07:09.0234150Z Running tests...
2023-01-11T22:07:09.0234397Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0234709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0234973Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0235236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29964
2023-01-11T22:07:09.0235460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29965
2023-01-11T22:07:09.0235827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0236002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0236385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0236557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0236918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0237142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0237523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0237707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0237953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0238194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0238594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0238986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0239198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0239629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0239880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0240115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0240216Z ok (6.244s)
2023-01-11T22:07:09.0240238Z 
2023-01-11T22:07:09.0240506Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0240616Z Ran 1 test in 6.244s
2023-01-11T22:07:09.0240636Z 
2023-01-11T22:07:09.0240731Z OK
2023-01-11T22:07:09.0240750Z 
2023-01-11T22:07:09.0240857Z Generating XML reports...
2023-01-11T22:07:09.0241307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215215.xml
2023-01-11T22:07:09.0241679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0241854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0242235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0242423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0242443Z 
2023-01-11T22:07:09.0242550Z Running tests...
2023-01-11T22:07:09.0242813Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0243110Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0243466Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0243686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30082
2023-01-11T22:07:09.0243904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30083
2023-01-11T22:07:09.0244279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0244507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0244898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0245088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0245451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0245612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0245986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0246171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0246415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0246655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0247059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0247452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0247677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0247909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0248127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0248361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0248629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0248866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0248971Z ok (6.336s)
2023-01-11T22:07:09.0248991Z 
2023-01-11T22:07:09.0249258Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0249369Z Ran 1 test in 6.337s
2023-01-11T22:07:09.0249389Z 
2023-01-11T22:07:09.0249481Z OK
2023-01-11T22:07:09.0249500Z 
2023-01-11T22:07:09.0249605Z Generating XML reports...
2023-01-11T22:07:09.0250056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215223.xml
2023-01-11T22:07:09.0250427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0250603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0250978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0251167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0251187Z 
2023-01-11T22:07:09.0251295Z Running tests...
2023-01-11T22:07:09.0251554Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0251868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0252123Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0252935Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.624s)
2023-01-11T22:07:09.0252956Z 
2023-01-11T22:07:09.0253218Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0253329Z Ran 1 test in 1.624s
2023-01-11T22:07:09.0253348Z 
2023-01-11T22:07:09.0253455Z OK (skipped=1)
2023-01-11T22:07:09.0253475Z 
2023-01-11T22:07:09.0253639Z Generating XML reports...
2023-01-11T22:07:09.0254099Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215232.xml
2023-01-11T22:07:09.0254466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0254641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0255008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0255198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0255218Z 
2023-01-11T22:07:09.0255325Z Running tests...
2023-01-11T22:07:09.0255585Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0255896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0256188Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0256932Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2023-01-11T22:07:09.0256957Z 
2023-01-11T22:07:09.0257219Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0257331Z Ran 1 test in 1.634s
2023-01-11T22:07:09.0257351Z 
2023-01-11T22:07:09.0257457Z OK (skipped=1)
2023-01-11T22:07:09.0257476Z 
2023-01-11T22:07:09.0257581Z Generating XML reports...
2023-01-11T22:07:09.0258026Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215237.xml
2023-01-11T22:07:09.0258398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0258572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0258948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0259139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0259162Z 
2023-01-11T22:07:09.0259269Z Running tests...
2023-01-11T22:07:09.0259530Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0259842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0260115Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0260333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30268
2023-01-11T22:07:09.0260550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30269
2023-01-11T22:07:09.0260919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0261092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0261470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0261720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0262089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0262243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0262614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0262845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0263097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0263336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0263736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0264136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0264363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0264591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0264784Z <class 'torch.nn.parameter.Parameter'>
2023-01-11T22:07:09.0264885Z ok (7.056s)
2023-01-11T22:07:09.0264907Z 
2023-01-11T22:07:09.0265171Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0265281Z Ran 1 test in 7.056s
2023-01-11T22:07:09.0265301Z 
2023-01-11T22:07:09.0265392Z OK
2023-01-11T22:07:09.0265411Z 
2023-01-11T22:07:09.0265534Z Generating XML reports...
2023-01-11T22:07:09.0265982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215241.xml
2023-01-11T22:07:09.0266352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0266510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0266887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0267073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0267093Z 
2023-01-11T22:07:09.0267200Z Running tests...
2023-01-11T22:07:09.0267462Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0267775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0268080Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0268299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30383
2023-01-11T22:07:09.0268518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30384
2023-01-11T22:07:09.0268871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0269045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0269418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0269606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0269970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0270142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0270515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0270758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0270987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0271230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0271631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0272067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0272301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0272529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0272629Z ok (6.954s)
2023-01-11T22:07:09.0272649Z 
2023-01-11T22:07:09.0272912Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0273025Z Ran 1 test in 6.954s
2023-01-11T22:07:09.0273045Z 
2023-01-11T22:07:09.0273118Z OK
2023-01-11T22:07:09.0273137Z 
2023-01-11T22:07:09.0273261Z Generating XML reports...
2023-01-11T22:07:09.0273711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215250.xml
2023-01-11T22:07:09.0274081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0274259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0274639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0274828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0274848Z 
2023-01-11T22:07:09.0274955Z Running tests...
2023-01-11T22:07:09.0275218Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0275519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0275782Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0275998Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30498
2023-01-11T22:07:09.0276213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30499
2023-01-11T22:07:09.0276580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0276755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0277129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0277316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0277661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0277838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0278210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0278395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0278638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0278881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0279279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0279984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0280299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0280510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0280744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0280974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0281075Z ok (7.457s)
2023-01-11T22:07:09.0281096Z 
2023-01-11T22:07:09.0281436Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0281557Z Ran 1 test in 7.458s
2023-01-11T22:07:09.0281577Z 
2023-01-11T22:07:09.0281670Z OK
2023-01-11T22:07:09.0281689Z 
2023-01-11T22:07:09.0281814Z Generating XML reports...
2023-01-11T22:07:09.0282256Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215300.xml
2023-01-11T22:07:09.0282625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0282805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0283186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0283375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0283395Z 
2023-01-11T22:07:09.0283501Z Running tests...
2023-01-11T22:07:09.0283761Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0284076Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0284368Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0284571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30617
2023-01-11T22:07:09.0284790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30618
2023-01-11T22:07:09.0285159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0285333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0285707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0285895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0286258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0286429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0286784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0286977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0287225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0287470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0287870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0288265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0288497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0288725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0289509Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:07:09.0289671Z ok (6.339s)
2023-01-11T22:07:09.0289692Z 
2023-01-11T22:07:09.0289959Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0290054Z Ran 1 test in 6.339s
2023-01-11T22:07:09.0290115Z 
2023-01-11T22:07:09.0290211Z OK
2023-01-11T22:07:09.0290231Z 
2023-01-11T22:07:09.0290355Z Generating XML reports...
2023-01-11T22:07:09.0290810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215310.xml
2023-01-11T22:07:09.0291176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0291357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0291738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0291931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0291951Z 
2023-01-11T22:07:09.0292063Z Running tests...
2023-01-11T22:07:09.0292306Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0292623Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0292899Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0293649Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.617s)
2023-01-11T22:07:09.0293674Z 
2023-01-11T22:07:09.0293934Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0294045Z Ran 1 test in 1.617s
2023-01-11T22:07:09.0294064Z 
2023-01-11T22:07:09.0294170Z OK (skipped=1)
2023-01-11T22:07:09.0294190Z 
2023-01-11T22:07:09.0294312Z Generating XML reports...
2023-01-11T22:07:09.0294757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215319.xml
2023-01-11T22:07:09.0295108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0295283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0295657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0295850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0295870Z 
2023-01-11T22:07:09.0295977Z Running tests...
2023-01-11T22:07:09.0296236Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0296547Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0296874Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2023-01-11T22:07:09.0296894Z 
2023-01-11T22:07:09.0297156Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0297252Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0297271Z 
2023-01-11T22:07:09.0297376Z OK (skipped=1)
2023-01-11T22:07:09.0297394Z 
2023-01-11T22:07:09.0297516Z Generating XML reports...
2023-01-11T22:07:09.0297966Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215323.xml
2023-01-11T22:07:09.0298396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0298567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0298946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0299134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0299153Z 
2023-01-11T22:07:09.0299262Z Running tests...
2023-01-11T22:07:09.0299551Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0299870Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0300115Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0300858Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2023-01-11T22:07:09.0300882Z 
2023-01-11T22:07:09.0301138Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0301249Z Ran 1 test in 1.627s
2023-01-11T22:07:09.0301268Z 
2023-01-11T22:07:09.0301377Z OK (skipped=1)
2023-01-11T22:07:09.0301396Z 
2023-01-11T22:07:09.0301517Z Generating XML reports...
2023-01-11T22:07:09.0301962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215325.xml
2023-01-11T22:07:09.0302313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0302487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0302863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0303055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0303074Z 
2023-01-11T22:07:09.0303181Z Running tests...
2023-01-11T22:07:09.0303442Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0303753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0304022Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0304246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30836
2023-01-11T22:07:09.0304448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30837
2023-01-11T22:07:09.0304816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0304988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0305358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0305530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0305908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0306097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0306476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0306648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0306891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0307135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0307597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0307991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0308218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0308446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0309280Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T22:07:09.0309620Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T22:07:09.0310413Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T22:07:09.0310737Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T22:07:09.0310822Z ok (7.451s)
2023-01-11T22:07:09.0310843Z 
2023-01-11T22:07:09.0311106Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0311218Z Ran 1 test in 7.452s
2023-01-11T22:07:09.0311238Z 
2023-01-11T22:07:09.0311329Z OK
2023-01-11T22:07:09.0311349Z 
2023-01-11T22:07:09.0311471Z Generating XML reports...
2023-01-11T22:07:09.0311922Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215329.xml
2023-01-11T22:07:09.0312298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0312471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0312832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0313022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0313043Z 
2023-01-11T22:07:09.0313151Z Running tests...
2023-01-11T22:07:09.0313412Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0313724Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0313995Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0314739Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T22:07:09.0314763Z 
2023-01-11T22:07:09.0315029Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0315141Z Ran 1 test in 1.634s
2023-01-11T22:07:09.0315161Z 
2023-01-11T22:07:09.0315267Z OK (skipped=1)
2023-01-11T22:07:09.0315286Z 
2023-01-11T22:07:09.0315391Z Generating XML reports...
2023-01-11T22:07:09.0315841Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215339.xml
2023-01-11T22:07:09.0316211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0316386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0316765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0317011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0317031Z 
2023-01-11T22:07:09.0317140Z Running tests...
2023-01-11T22:07:09.0317401Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0317713Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0318007Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0318763Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2023-01-11T22:07:09.0318784Z 
2023-01-11T22:07:09.0319051Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0319162Z Ran 1 test in 1.638s
2023-01-11T22:07:09.0319182Z 
2023-01-11T22:07:09.0319288Z OK (skipped=1)
2023-01-11T22:07:09.0319307Z 
2023-01-11T22:07:09.0319652Z Generating XML reports...
2023-01-11T22:07:09.0320115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215344.xml
2023-01-11T22:07:09.0320485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0320664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0321031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0321222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0321242Z 
2023-01-11T22:07:09.0321351Z Running tests...
2023-01-11T22:07:09.0321614Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0321927Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0322222Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0322443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31053
2023-01-11T22:07:09.0322659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31054
2023-01-11T22:07:09.0323033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0323190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0323571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0323760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0324126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0324296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0324666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0324854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0325104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0325331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0325732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0326126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0326440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0326682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0326905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0327141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0327595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0327998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0328233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0328449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0328685Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0328916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0329018Z ok (7.750s)
2023-01-11T22:07:09.0329038Z 
2023-01-11T22:07:09.0329303Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0329415Z Ran 1 test in 7.751s
2023-01-11T22:07:09.0329434Z 
2023-01-11T22:07:09.0329526Z OK
2023-01-11T22:07:09.0329548Z 
2023-01-11T22:07:09.0329672Z Generating XML reports...
2023-01-11T22:07:09.0330107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215348.xml
2023-01-11T22:07:09.0330479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0330652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0331034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0331225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0331245Z 
2023-01-11T22:07:09.0331353Z Running tests...
2023-01-11T22:07:09.0331612Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0331926Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0332186Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0332407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31174
2023-01-11T22:07:09.0332625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31175
2023-01-11T22:07:09.0332992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0333168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0333544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0333735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0334096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0334273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0334629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0334818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0335061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0335364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0335766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0336159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0336389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0336710Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:07:09.0336940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0337192Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:07:09.0337431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0337660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0337891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0338123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0338399Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:07:09.0338677Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:07:09.0338948Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:07:09.0339216Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:07:09.0339438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0339671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0339901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0340124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0340397Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:07:09.0340671Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:07:09.0340946Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T22:07:09.0341213Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T22:07:09.0341450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0341665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0341891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0342120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0342222Z ok (8.154s)
2023-01-11T22:07:09.0342243Z 
2023-01-11T22:07:09.0342518Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0342631Z Ran 1 test in 8.154s
2023-01-11T22:07:09.0342651Z 
2023-01-11T22:07:09.0342741Z OK
2023-01-11T22:07:09.0342760Z 
2023-01-11T22:07:09.0342883Z Generating XML reports...
2023-01-11T22:07:09.0343317Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215358.xml
2023-01-11T22:07:09.0343688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0343927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0344308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0344498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0344519Z 
2023-01-11T22:07:09.0344628Z Running tests...
2023-01-11T22:07:09.0344945Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0345268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0345519Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0346267Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2023-01-11T22:07:09.0346310Z 
2023-01-11T22:07:09.0346552Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0346664Z Ran 1 test in 1.632s
2023-01-11T22:07:09.0346683Z 
2023-01-11T22:07:09.0346791Z OK (skipped=1)
2023-01-11T22:07:09.0346810Z 
2023-01-11T22:07:09.0346934Z Generating XML reports...
2023-01-11T22:07:09.0347383Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215409.xml
2023-01-11T22:07:09.0347756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0347929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0348311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0348487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0348525Z 
2023-01-11T22:07:09.0348660Z Running tests...
2023-01-11T22:07:09.0348925Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0349239Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0349511Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0349731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31327
2023-01-11T22:07:09.0349948Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31328
2023-01-11T22:07:09.0350319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0350495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0350861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0351050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0351411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0351583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0351956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0352146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0352393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0352793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0353078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0353472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0353700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0354295Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:07:09.0354523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0355058Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:07:09.0355299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0355527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0355804Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T22:07:09.0356074Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T22:07:09.0356372Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T22:07:09.0356684Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T22:07:09.0357014Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T22:07:09.0357305Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T22:07:09.0357625Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T22:07:09.0357945Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T22:07:09.0358179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0358406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0358511Z ok (7.427s)
2023-01-11T22:07:09.0358531Z 
2023-01-11T22:07:09.0358804Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0358914Z Ran 1 test in 7.427s
2023-01-11T22:07:09.0358934Z 
2023-01-11T22:07:09.0359009Z OK
2023-01-11T22:07:09.0359028Z 
2023-01-11T22:07:09.0359151Z Generating XML reports...
2023-01-11T22:07:09.0359830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215413.xml
2023-01-11T22:07:09.0360223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0360400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0360782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0360974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0361068Z 
2023-01-11T22:07:09.0361186Z Running tests...
2023-01-11T22:07:09.0361434Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0361749Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0362139Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0362159Z 
2023-01-11T22:07:09.0362516Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0362640Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0362661Z 
2023-01-11T22:07:09.0362769Z OK (skipped=1)
2023-01-11T22:07:09.0362788Z 
2023-01-11T22:07:09.0362911Z Generating XML reports...
2023-01-11T22:07:09.0363365Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215423.xml
2023-01-11T22:07:09.0363741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0363914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0364276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0364466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0364486Z 
2023-01-11T22:07:09.0364593Z Running tests...
2023-01-11T22:07:09.0364856Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0365169Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0365563Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0365587Z 
2023-01-11T22:07:09.0365848Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0365961Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0365980Z 
2023-01-11T22:07:09.0366087Z OK (skipped=1)
2023-01-11T22:07:09.0366106Z 
2023-01-11T22:07:09.0366213Z Generating XML reports...
2023-01-11T22:07:09.0366660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215425.xml
2023-01-11T22:07:09.0367037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0367210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0367589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0367779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0367798Z 
2023-01-11T22:07:09.0367907Z Running tests...
2023-01-11T22:07:09.0368171Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0368465Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0368919Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0368939Z 
2023-01-11T22:07:09.0369198Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0369311Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0369331Z 
2023-01-11T22:07:09.0369439Z OK (skipped=1)
2023-01-11T22:07:09.0369458Z 
2023-01-11T22:07:09.0369581Z Generating XML reports...
2023-01-11T22:07:09.0370027Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215428.xml
2023-01-11T22:07:09.0370461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0370637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0371016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0371191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0371210Z 
2023-01-11T22:07:09.0371318Z Running tests...
2023-01-11T22:07:09.0371617Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0371940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0372387Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0372412Z 
2023-01-11T22:07:09.0372670Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0372782Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0372801Z 
2023-01-11T22:07:09.0372909Z OK (skipped=1)
2023-01-11T22:07:09.0372928Z 
2023-01-11T22:07:09.0373050Z Generating XML reports...
2023-01-11T22:07:09.0373482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215430.xml
2023-01-11T22:07:09.0373857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0374032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0374416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0374607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0374629Z 
2023-01-11T22:07:09.0374738Z Running tests...
2023-01-11T22:07:09.0374999Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0375311Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0375760Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0375781Z 
2023-01-11T22:07:09.0376044Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0376140Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0376159Z 
2023-01-11T22:07:09.0376265Z OK (skipped=1)
2023-01-11T22:07:09.0376284Z 
2023-01-11T22:07:09.0376406Z Generating XML reports...
2023-01-11T22:07:09.0376853Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215432.xml
2023-01-11T22:07:09.0377227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0377400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0377781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0377971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0377990Z 
2023-01-11T22:07:09.0378081Z Running tests...
2023-01-11T22:07:09.0378341Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0378652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0379096Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0379169Z 
2023-01-11T22:07:09.0379431Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0379542Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0379561Z 
2023-01-11T22:07:09.0379669Z OK (skipped=1)
2023-01-11T22:07:09.0379688Z 
2023-01-11T22:07:09.0379811Z Generating XML reports...
2023-01-11T22:07:09.0380255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215435.xml
2023-01-11T22:07:09.0380652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0380835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0381214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0381404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0381427Z 
2023-01-11T22:07:09.0381535Z Running tests...
2023-01-11T22:07:09.0381797Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0382109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0382557Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0382580Z 
2023-01-11T22:07:09.0382842Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0382953Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0382972Z 
2023-01-11T22:07:09.0383063Z OK (skipped=1)
2023-01-11T22:07:09.0383083Z 
2023-01-11T22:07:09.0383205Z Generating XML reports...
2023-01-11T22:07:09.0383655Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215437.xml
2023-01-11T22:07:09.0384031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0384205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0384582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0384773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0384793Z 
2023-01-11T22:07:09.0384904Z Running tests...
2023-01-11T22:07:09.0385148Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0385458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0385901Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0385925Z 
2023-01-11T22:07:09.0386181Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0386292Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0386311Z 
2023-01-11T22:07:09.0386417Z OK (skipped=1)
2023-01-11T22:07:09.0386436Z 
2023-01-11T22:07:09.0386557Z Generating XML reports...
2023-01-11T22:07:09.0387004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215440.xml
2023-01-11T22:07:09.0387368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0387543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0387904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0388153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0388173Z 
2023-01-11T22:07:09.0388280Z Running tests...
2023-01-11T22:07:09.0398963Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0399689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0400353Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0400380Z 
2023-01-11T22:07:09.0400689Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0400809Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0400831Z 
2023-01-11T22:07:09.0400942Z OK (skipped=1)
2023-01-11T22:07:09.0400962Z 
2023-01-11T22:07:09.0401072Z Generating XML reports...
2023-01-11T22:07:09.0401562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215442.xml
2023-01-11T22:07:09.0401953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0402137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0402539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0402741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0402761Z 
2023-01-11T22:07:09.0402872Z Running tests...
2023-01-11T22:07:09.0403150Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0403467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0403946Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0403986Z 
2023-01-11T22:07:09.0404245Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0404362Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0404381Z 
2023-01-11T22:07:09.0404493Z OK (skipped=1)
2023-01-11T22:07:09.0404513Z 
2023-01-11T22:07:09.0404639Z Generating XML reports...
2023-01-11T22:07:09.0405119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215444.xml
2023-01-11T22:07:09.0405511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0405696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0406095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0406281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0406301Z 
2023-01-11T22:07:09.0406411Z Running tests...
2023-01-11T22:07:09.0406688Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0407020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0407436Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0407457Z 
2023-01-11T22:07:09.0407729Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0407843Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0407863Z 
2023-01-11T22:07:09.0407970Z OK (skipped=1)
2023-01-11T22:07:09.0407989Z 
2023-01-11T22:07:09.0408114Z Generating XML reports...
2023-01-11T22:07:09.0408667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215447.xml
2023-01-11T22:07:09.0409058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0409240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0409642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0409887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0409909Z 
2023-01-11T22:07:09.0410024Z Running tests...
2023-01-11T22:07:09.0410305Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0410635Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0411049Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:07:09.0411075Z 
2023-01-11T22:07:09.0411338Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0411452Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0411471Z 
2023-01-11T22:07:09.0411581Z OK (skipped=1)
2023-01-11T22:07:09.0411600Z 
2023-01-11T22:07:09.0411726Z Generating XML reports...
2023-01-11T22:07:09.0412201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215449.xml
2023-01-11T22:07:09.0412592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0412774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0413175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0413377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0413397Z 
2023-01-11T22:07:09.0413491Z Running tests...
2023-01-11T22:07:09.0413769Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0414099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0414373Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0415167Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2023-01-11T22:07:09.0415188Z 
2023-01-11T22:07:09.0415464Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0415579Z Ran 1 test in 1.646s
2023-01-11T22:07:09.0415602Z 
2023-01-11T22:07:09.0415710Z OK (skipped=1)
2023-01-11T22:07:09.0415730Z 
2023-01-11T22:07:09.0415855Z Generating XML reports...
2023-01-11T22:07:09.0416311Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215452.xml
2023-01-11T22:07:09.0416699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0416880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0417286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0417482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0417502Z 
2023-01-11T22:07:09.0417611Z Running tests...
2023-01-11T22:07:09.0417884Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0418216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0418549Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0418763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31876
2023-01-11T22:07:09.0418992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31877
2023-01-11T22:07:09.0419389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0419630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0420045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0420242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0420633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0420819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0421202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0421397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0421650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0421907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0422324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0422742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0422983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0423226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0423328Z ok (6.649s)
2023-01-11T22:07:09.0423349Z 
2023-01-11T22:07:09.0423609Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0423724Z Ran 1 test in 6.650s
2023-01-11T22:07:09.0423744Z 
2023-01-11T22:07:09.0423835Z OK
2023-01-11T22:07:09.0423855Z 
2023-01-11T22:07:09.0423981Z Generating XML reports...
2023-01-11T22:07:09.0424458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215456.xml
2023-01-11T22:07:09.0424845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0425027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0425430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0425631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0425652Z 
2023-01-11T22:07:09.0425744Z Running tests...
2023-01-11T22:07:09.0426022Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0426353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0426639Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0426872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31990
2023-01-11T22:07:09.0427099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31991
2023-01-11T22:07:09.0427488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0427671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0428134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0428334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0428719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0428901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0429354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0429561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0429819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0430076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0430502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0430900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0431142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0431382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0431631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0431875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0432291Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpe_diiu0s'>
2023-01-11T22:07:09.0432461Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:07:09.0432876Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp3p3c1525'>
2023-01-11T22:07:09.0433029Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:07:09.0433132Z ok (7.548s)
2023-01-11T22:07:09.0433152Z 
2023-01-11T22:07:09.0433427Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0433541Z Ran 1 test in 7.548s
2023-01-11T22:07:09.0433561Z 
2023-01-11T22:07:09.0433653Z OK
2023-01-11T22:07:09.0433671Z 
2023-01-11T22:07:09.0433796Z Generating XML reports...
2023-01-11T22:07:09.0434274Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215505.xml
2023-01-11T22:07:09.0434664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0434847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0435232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0435431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0435452Z 
2023-01-11T22:07:09.0435560Z Running tests...
2023-01-11T22:07:09.0435834Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0436163Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0436451Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.005s)
2023-01-11T22:07:09.0436471Z 
2023-01-11T22:07:09.0436746Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0436860Z Ran 1 test in 0.005s
2023-01-11T22:07:09.0436880Z 
2023-01-11T22:07:09.0436988Z OK (skipped=1)
2023-01-11T22:07:09.0437009Z 
2023-01-11T22:07:09.0437119Z Generating XML reports...
2023-01-11T22:07:09.0437589Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215515.xml
2023-01-11T22:07:09.0438047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0438230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0438630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0438827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0438892Z 
2023-01-11T22:07:09.0439011Z Running tests...
2023-01-11T22:07:09.0439291Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0439795Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0440075Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0440310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32142
2023-01-11T22:07:09.0440541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32143
2023-01-11T22:07:09.0440935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0441118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0441526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0441724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0442113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0442279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0442678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0442879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0443133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0443388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0443805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0444223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0444464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0444687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0444931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0445178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0445279Z ok (7.346s)
2023-01-11T22:07:09.0445299Z 
2023-01-11T22:07:09.0445576Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0445690Z Ran 1 test in 7.346s
2023-01-11T22:07:09.0445710Z 
2023-01-11T22:07:09.0445801Z OK
2023-01-11T22:07:09.0445820Z 
2023-01-11T22:07:09.0445944Z Generating XML reports...
2023-01-11T22:07:09.0446423Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215517.xml
2023-01-11T22:07:09.0446797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0446980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0447380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0447667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0447687Z 
2023-01-11T22:07:09.0447796Z Running tests...
2023-01-11T22:07:09.0448076Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0448410Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0448810Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0449034Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32261
2023-01-11T22:07:09.0449266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32262
2023-01-11T22:07:09.0449664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0449846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0450252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0450449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0450835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0451015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0451420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0451601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0451853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0452105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0452532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0452949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0453189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0453428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0453683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0453937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0454338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0454755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0455012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:09.0455264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:09.0455686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0456105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0456208Z ok (5.851s)
2023-01-11T22:07:09.0456227Z 
2023-01-11T22:07:09.0456504Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0456619Z Ran 1 test in 5.852s
2023-01-11T22:07:09.0456639Z 
2023-01-11T22:07:09.0456716Z OK
2023-01-11T22:07:09.0456735Z 
2023-01-11T22:07:09.0456928Z Generating XML reports...
2023-01-11T22:07:09.0457409Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215527.xml
2023-01-11T22:07:09.0457800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0457983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0458381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0458625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0458646Z 
2023-01-11T22:07:09.0458763Z Running tests...
2023-01-11T22:07:09.0459028Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0459358Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0459652Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0459882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32383
2023-01-11T22:07:09.0460111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32384
2023-01-11T22:07:09.0460501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0460682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0461088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0461285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0461657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0461839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0462240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0462436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0462691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0462948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0463369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0463787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0464028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0464253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0464509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0464759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0465177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0465592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0465847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:09.0466098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:09.0466516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0466999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0467088Z ok (16.656s)
2023-01-11T22:07:09.0467108Z 
2023-01-11T22:07:09.0467385Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0467502Z Ran 1 test in 16.657s
2023-01-11T22:07:09.0467522Z 
2023-01-11T22:07:09.0467616Z OK
2023-01-11T22:07:09.0467635Z 
2023-01-11T22:07:09.0467761Z Generating XML reports...
2023-01-11T22:07:09.0468277Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215536.xml
2023-01-11T22:07:09.0468680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0468861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0469247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0469453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0469473Z 
2023-01-11T22:07:09.0469583Z Running tests...
2023-01-11T22:07:09.0469859Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0470185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0470508Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0470741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32505
2023-01-11T22:07:09.0470964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32506
2023-01-11T22:07:09.0471355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0471521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0471926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0472122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0472510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0472689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0473091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0473289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0473543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0473797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0474205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0474621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0474862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0475104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0475206Z ok (7.648s)
2023-01-11T22:07:09.0475230Z 
2023-01-11T22:07:09.0475507Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0475620Z Ran 1 test in 7.648s
2023-01-11T22:07:09.0475640Z 
2023-01-11T22:07:09.0475731Z OK
2023-01-11T22:07:09.0475751Z 
2023-01-11T22:07:09.0475862Z Generating XML reports...
2023-01-11T22:07:09.0476336Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215555.xml
2023-01-11T22:07:09.0476794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0476978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0477378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0477574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0477594Z 
2023-01-11T22:07:09.0477746Z Running tests...
2023-01-11T22:07:09.0478029Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0478360Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0478650Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0478880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32624
2023-01-11T22:07:09.0479112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32625
2023-01-11T22:07:09.0479835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0480013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0480406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0480604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0480972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0481126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0481504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0481695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0481939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0482184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0482582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0482981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0483210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0483438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0483524Z ok (7.631s)
2023-01-11T22:07:09.0483545Z 
2023-01-11T22:07:09.0483816Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0483931Z Ran 1 test in 7.632s
2023-01-11T22:07:09.0483951Z 
2023-01-11T22:07:09.0484044Z OK
2023-01-11T22:07:09.0484063Z 
2023-01-11T22:07:09.0484187Z Generating XML reports...
2023-01-11T22:07:09.0484638Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215605.xml
2023-01-11T22:07:09.0485008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0485186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0485565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0485740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0485760Z 
2023-01-11T22:07:09.0485868Z Running tests...
2023-01-11T22:07:09.0486224Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0486541Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0486798Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0487015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32743
2023-01-11T22:07:09.0487232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32744
2023-01-11T22:07:09.0487669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0487838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0488211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0488384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0488765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0488953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0489324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0489510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0489756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0489981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0490381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0490778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0491008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0491233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0491333Z ok (7.441s)
2023-01-11T22:07:09.0491353Z 
2023-01-11T22:07:09.0491614Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0491726Z Ran 1 test in 7.442s
2023-01-11T22:07:09.0491745Z 
2023-01-11T22:07:09.0491837Z OK
2023-01-11T22:07:09.0491856Z 
2023-01-11T22:07:09.0491966Z Generating XML reports...
2023-01-11T22:07:09.0492415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215615.xml
2023-01-11T22:07:09.0492782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0492955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0493335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0493523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0493543Z 
2023-01-11T22:07:09.0493651Z Running tests...
2023-01-11T22:07:09.0493909Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0494220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0494470Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0494687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32858
2023-01-11T22:07:09.0494901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32859
2023-01-11T22:07:09.0495268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0495499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0495882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0496071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0496433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0496631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0497015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0497199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0497445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0497690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0498087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0498481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0498709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0498939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0499729Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:07:09.0500517Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:07:09.0500757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0500993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0501078Z ok (7.345s)
2023-01-11T22:07:09.0501100Z 
2023-01-11T22:07:09.0501367Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0501478Z Ran 1 test in 7.345s
2023-01-11T22:07:09.0501497Z 
2023-01-11T22:07:09.0501589Z OK
2023-01-11T22:07:09.0501608Z 
2023-01-11T22:07:09.0501732Z Generating XML reports...
2023-01-11T22:07:09.0502185Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215625.xml
2023-01-11T22:07:09.0502556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0502730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0503092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0503280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0503300Z 
2023-01-11T22:07:09.0503465Z Running tests...
2023-01-11T22:07:09.0503728Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0504043Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0504323Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0505117Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.623s)
2023-01-11T22:07:09.0505139Z 
2023-01-11T22:07:09.0505404Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0505518Z Ran 1 test in 1.623s
2023-01-11T22:07:09.0505536Z 
2023-01-11T22:07:09.0505644Z OK (skipped=1)
2023-01-11T22:07:09.0505663Z 
2023-01-11T22:07:09.0505774Z Generating XML reports...
2023-01-11T22:07:09.0506226Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215635.xml
2023-01-11T22:07:09.0506593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0506768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0507146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0507339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0507359Z 
2023-01-11T22:07:09.0507466Z Running tests...
2023-01-11T22:07:09.0507727Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0508021Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0508300Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0509050Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2023-01-11T22:07:09.0509071Z 
2023-01-11T22:07:09.0509333Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0509448Z Ran 1 test in 1.635s
2023-01-11T22:07:09.0509468Z 
2023-01-11T22:07:09.0509573Z OK (skipped=1)
2023-01-11T22:07:09.0509592Z 
2023-01-11T22:07:09.0509712Z Generating XML reports...
2023-01-11T22:07:09.0510159Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215639.xml
2023-01-11T22:07:09.0510529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0510707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0511071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0511258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0511277Z 
2023-01-11T22:07:09.0511383Z Running tests...
2023-01-11T22:07:09.0511640Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0511952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0512228Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0512446Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33045
2023-01-11T22:07:09.0512664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33046
2023-01-11T22:07:09.0513095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0513255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0513631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0513821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0514223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0514403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0514779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0514966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0515217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0515444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0515842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0516233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0516465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0516692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0517028Z STAGE:2023-01-11 21:56:49 33045:33045 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0517348Z STAGE:2023-01-11 21:56:49 33046:33046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0517590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0517818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0518136Z STAGE:2023-01-11 21:56:50 33045:33045 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0518459Z STAGE:2023-01-11 21:56:50 33046:33046 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0518806Z STAGE:2023-01-11 21:56:50 33045:33045 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0519149Z STAGE:2023-01-11 21:56:50 33046:33046 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0520171Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:07:09.0520973Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:07:09.0521516Z STAGE:2023-01-11 21:56:50 33045:33045 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:56:50 33046:33046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0521641Z 
2023-01-11T22:07:09.0522197Z STAGE:2023-01-11 21:56:50 33046:33046 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:56:50 33045:33045 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0522217Z 
2023-01-11T22:07:09.0522830Z STAGE:2023-01-11 21:56:50 33046:33046 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:56:50 33045:33045 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0522852Z 
2023-01-11T22:07:09.0522958Z ok (6.736s)
2023-01-11T22:07:09.0522978Z 
2023-01-11T22:07:09.0523242Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0523355Z Ran 1 test in 6.736s
2023-01-11T22:07:09.0523374Z 
2023-01-11T22:07:09.0523467Z OK
2023-01-11T22:07:09.0523486Z 
2023-01-11T22:07:09.0523611Z Generating XML reports...
2023-01-11T22:07:09.0524069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215644.xml
2023-01-11T22:07:09.0524422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0524596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0524975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0525168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0525188Z 
2023-01-11T22:07:09.0525293Z Running tests...
2023-01-11T22:07:09.0525550Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0525861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0526129Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0526335Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33167
2023-01-11T22:07:09.0526553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33168
2023-01-11T22:07:09.0526924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0527097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0527476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0527665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0528030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0528202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0528569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0528743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0528991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0529232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0529634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0530032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0530260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0530489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0530649Z ok (6.955s)
2023-01-11T22:07:09.0530669Z 
2023-01-11T22:07:09.0530935Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0531031Z Ran 1 test in 6.955s
2023-01-11T22:07:09.0531051Z 
2023-01-11T22:07:09.0531143Z OK
2023-01-11T22:07:09.0531162Z 
2023-01-11T22:07:09.0531285Z Generating XML reports...
2023-01-11T22:07:09.0531737Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215653.xml
2023-01-11T22:07:09.0532151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0532335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0532717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0532906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0532929Z 
2023-01-11T22:07:09.0533021Z Running tests...
2023-01-11T22:07:09.0533283Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0533593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0533868Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0534622Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2023-01-11T22:07:09.0534642Z 
2023-01-11T22:07:09.0534900Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0535010Z Ran 1 test in 1.645s
2023-01-11T22:07:09.0535029Z 
2023-01-11T22:07:09.0535134Z OK (skipped=1)
2023-01-11T22:07:09.0535157Z 
2023-01-11T22:07:09.0535279Z Generating XML reports...
2023-01-11T22:07:09.0535725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215702.xml
2023-01-11T22:07:09.0536078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0536253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0536634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0536823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0536842Z 
2023-01-11T22:07:09.0536948Z Running tests...
2023-01-11T22:07:09.0537268Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0537583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0537866Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0538067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33316
2023-01-11T22:07:09.0538286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33317
2023-01-11T22:07:09.0538654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0538826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0539205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0539393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0539755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0539926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0540379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0540540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0540785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0541027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0541470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0541876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0542107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0542336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0543257Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:07:09.0543369Z   warnings.warn(
2023-01-11T22:07:09.0544279Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:07:09.0544391Z   warnings.warn(
2023-01-11T22:07:09.0544616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0544842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0544941Z ok (6.345s)
2023-01-11T22:07:09.0544960Z 
2023-01-11T22:07:09.0545221Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0545331Z Ran 1 test in 6.346s
2023-01-11T22:07:09.0545350Z 
2023-01-11T22:07:09.0545440Z OK
2023-01-11T22:07:09.0545459Z 
2023-01-11T22:07:09.0545581Z Generating XML reports...
2023-01-11T22:07:09.0546033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215707.xml
2023-01-11T22:07:09.0546387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0546563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0546937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0547128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0547148Z 
2023-01-11T22:07:09.0547255Z Running tests...
2023-01-11T22:07:09.0547513Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0547825Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0548102Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0548894Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2023-01-11T22:07:09.0548916Z 
2023-01-11T22:07:09.0549220Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0549333Z Ran 1 test in 1.648s
2023-01-11T22:07:09.0549354Z 
2023-01-11T22:07:09.0549462Z OK (skipped=1)
2023-01-11T22:07:09.0549480Z 
2023-01-11T22:07:09.0549602Z Generating XML reports...
2023-01-11T22:07:09.0550049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215716.xml
2023-01-11T22:07:09.0550414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0550649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0551051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0551235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0551253Z 
2023-01-11T22:07:09.0551345Z Running tests...
2023-01-11T22:07:09.0551610Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0551922Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0552192Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0552410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33468
2023-01-11T22:07:09.0552626Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33469
2023-01-11T22:07:09.0553001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0553173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0553534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0553725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0554091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0554261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0554632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0554816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0555064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0555306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0555704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0556084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0556318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0556543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0556876Z STAGE:2023-01-11 21:57:25 33469:33469 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0557195Z STAGE:2023-01-11 21:57:25 33468:33468 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0557434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0557669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:07:09.0558003Z STAGE:2023-01-11 21:57:25 33469:33469 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0558336Z STAGE:2023-01-11 21:57:25 33468:33468 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0558726Z STAGE:2023-01-11 21:57:25 33469:33469 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0559072Z STAGE:2023-01-11 21:57:25 33468:33468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0559622Z STAGE:2023-01-11 21:57:25 33468:33468 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0559973Z STAGE:2023-01-11 21:57:26 33468:33468 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0560387Z STAGE:2023-01-11 21:57:26 33468:33468 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0560498Z ok (7.154s)
2023-01-11T22:07:09.0560519Z 
2023-01-11T22:07:09.0560783Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0560895Z Ran 1 test in 7.154s
2023-01-11T22:07:09.0560914Z 
2023-01-11T22:07:09.0561007Z OK
2023-01-11T22:07:09.0561026Z 
2023-01-11T22:07:09.0561138Z Generating XML reports...
2023-01-11T22:07:09.0561593Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215720.xml
2023-01-11T22:07:09.0561965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0562138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0562518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0562712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0562732Z 
2023-01-11T22:07:09.0562839Z Running tests...
2023-01-11T22:07:09.0563101Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0563398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0563662Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0563885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33590
2023-01-11T22:07:09.0564100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33591
2023-01-11T22:07:09.0564467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0564640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0565020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0565210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0565569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0565725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0566101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0566286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0566530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0566771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0567172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0567568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0567795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0568007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0568183Z ok (6.845s)
2023-01-11T22:07:09.0568202Z 
2023-01-11T22:07:09.0568470Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0568581Z Ran 1 test in 6.845s
2023-01-11T22:07:09.0568600Z 
2023-01-11T22:07:09.0568693Z OK
2023-01-11T22:07:09.0568712Z 
2023-01-11T22:07:09.0568835Z Generating XML reports...
2023-01-11T22:07:09.0569285Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215729.xml
2023-01-11T22:07:09.0569699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0569881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0570246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0571916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0571942Z 
2023-01-11T22:07:09.0572053Z Running tests...
2023-01-11T22:07:09.0572318Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0572634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0572908Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0573130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33705
2023-01-11T22:07:09.0573349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33706
2023-01-11T22:07:09.0573717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0573875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0574254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0574449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0574813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0574986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0575361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0575556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0575803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0576030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0576431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0576830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0577058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0577287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0577389Z ok (6.951s)
2023-01-11T22:07:09.0577409Z 
2023-01-11T22:07:09.0577675Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0577789Z Ran 1 test in 6.951s
2023-01-11T22:07:09.0577808Z 
2023-01-11T22:07:09.0577898Z OK
2023-01-11T22:07:09.0577917Z 
2023-01-11T22:07:09.0578023Z Generating XML reports...
2023-01-11T22:07:09.0578471Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215739.xml
2023-01-11T22:07:09.0578840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0579084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0579466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0579656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0579675Z 
2023-01-11T22:07:09.0579785Z Running tests...
2023-01-11T22:07:09.0580045Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0580383Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0580668Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0581416Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2023-01-11T22:07:09.0581440Z 
2023-01-11T22:07:09.0581701Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0581815Z Ran 1 test in 1.636s
2023-01-11T22:07:09.0581834Z 
2023-01-11T22:07:09.0581945Z OK (skipped=1)
2023-01-11T22:07:09.0581964Z 
2023-01-11T22:07:09.0582087Z Generating XML reports...
2023-01-11T22:07:09.0582537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215748.xml
2023-01-11T22:07:09.0582907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0583081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0583442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0583637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0583656Z 
2023-01-11T22:07:09.0583764Z Running tests...
2023-01-11T22:07:09.0584020Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0584333Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0584593Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0585338Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2023-01-11T22:07:09.0585359Z 
2023-01-11T22:07:09.0585619Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0585737Z Ran 1 test in 1.621s
2023-01-11T22:07:09.0585756Z 
2023-01-11T22:07:09.0585863Z OK (skipped=1)
2023-01-11T22:07:09.0585882Z 
2023-01-11T22:07:09.0585986Z Generating XML reports...
2023-01-11T22:07:09.0586436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215753.xml
2023-01-11T22:07:09.0586808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0586984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0587367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0587556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0587576Z 
2023-01-11T22:07:09.0587686Z Running tests...
2023-01-11T22:07:09.0587948Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0588302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0588594Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0589381Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.619s)
2023-01-11T22:07:09.0589403Z 
2023-01-11T22:07:09.0589671Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0589785Z Ran 1 test in 1.619s
2023-01-11T22:07:09.0589804Z 
2023-01-11T22:07:09.0589912Z OK (skipped=1)
2023-01-11T22:07:09.0589930Z 
2023-01-11T22:07:09.0590053Z Generating XML reports...
2023-01-11T22:07:09.0590500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215757.xml
2023-01-11T22:07:09.0590876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0591052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0591411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0591605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0591624Z 
2023-01-11T22:07:09.0591736Z Running tests...
2023-01-11T22:07:09.0592000Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0592315Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0592611Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0592838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33922
2023-01-11T22:07:09.0593055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33923
2023-01-11T22:07:09.0593427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0593587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0593968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0594159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0594524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0594697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0595066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0595259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0595507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0595733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0596136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0596536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0596768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0597001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0597105Z ok (7.446s)
2023-01-11T22:07:09.0597175Z 
2023-01-11T22:07:09.0597442Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0597555Z Ran 1 test in 7.446s
2023-01-11T22:07:09.0597574Z 
2023-01-11T22:07:09.0597665Z OK
2023-01-11T22:07:09.0597684Z 
2023-01-11T22:07:09.0597790Z Generating XML reports...
2023-01-11T22:07:09.0598243Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215801.xml
2023-01-11T22:07:09.0598659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0598842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0599224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0599640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0599662Z 
2023-01-11T22:07:09.0599784Z Running tests...
2023-01-11T22:07:09.0600049Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0600346Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0600619Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0600837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34041
2023-01-11T22:07:09.0601051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34042
2023-01-11T22:07:09.0601421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0601597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0601975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0602163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0602530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0602688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0603062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0603248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0603497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0603740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0604143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0604541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0604776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0605155Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T22:07:09.0605393Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T22:07:09.0605624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0605999Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T22:07:09.0606251Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T22:07:09.0606355Z ok (6.941s)
2023-01-11T22:07:09.0606374Z 
2023-01-11T22:07:09.0606638Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0606751Z Ran 1 test in 6.941s
2023-01-11T22:07:09.0606844Z 
2023-01-11T22:07:09.0606942Z OK
2023-01-11T22:07:09.0606961Z 
2023-01-11T22:07:09.0607069Z Generating XML reports...
2023-01-11T22:07:09.0607527Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215811.xml
2023-01-11T22:07:09.0607891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0608065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0608500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0608698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0608717Z 
2023-01-11T22:07:09.0608828Z Running tests...
2023-01-11T22:07:09.0609091Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0609403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0609652Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0609871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34156
2023-01-11T22:07:09.0610090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34157
2023-01-11T22:07:09.0610460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0610638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0611020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0611211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0611568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0611728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0612100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0612286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0612531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0612780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0613180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0613580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0613812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0614044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0614265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0614506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0614903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0615300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0615404Z ok (4.141s)
2023-01-11T22:07:09.0615424Z 
2023-01-11T22:07:09.0615684Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0615794Z Ran 1 test in 4.141s
2023-01-11T22:07:09.0615814Z 
2023-01-11T22:07:09.0615907Z OK
2023-01-11T22:07:09.0615981Z 
2023-01-11T22:07:09.0616111Z Generating XML reports...
2023-01-11T22:07:09.0616544Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215820.xml
2023-01-11T22:07:09.0616914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0617089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0617525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0617724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0617743Z 
2023-01-11T22:07:09.0617852Z Running tests...
2023-01-11T22:07:09.0618115Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0618430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0618673Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0618896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34263
2023-01-11T22:07:09.0619111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34264
2023-01-11T22:07:09.0619481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0619655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0620036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0620226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0620596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0620767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0621129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0621316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0621563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0621805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0622209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0622602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0622828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0623069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0623294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0623518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0623916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0624313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0624416Z ok (4.232s)
2023-01-11T22:07:09.0624436Z 
2023-01-11T22:07:09.0624699Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0624811Z Ran 1 test in 4.232s
2023-01-11T22:07:09.0624830Z 
2023-01-11T22:07:09.0624923Z OK
2023-01-11T22:07:09.0624942Z 
2023-01-11T22:07:09.0625067Z Generating XML reports...
2023-01-11T22:07:09.0625497Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215827.xml
2023-01-11T22:07:09.0625935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0626110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0626486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0626721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0626741Z 
2023-01-11T22:07:09.0626854Z Running tests...
2023-01-11T22:07:09.0627119Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0627434Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0627712Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0628450Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2023-01-11T22:07:09.0628488Z 
2023-01-11T22:07:09.0628731Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0628844Z Ran 1 test in 1.644s
2023-01-11T22:07:09.0628863Z 
2023-01-11T22:07:09.0628974Z OK (skipped=1)
2023-01-11T22:07:09.0628994Z 
2023-01-11T22:07:09.0629118Z Generating XML reports...
2023-01-11T22:07:09.0629562Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215834.xml
2023-01-11T22:07:09.0629927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0630108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0630489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0630681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0630701Z 
2023-01-11T22:07:09.0630791Z Running tests...
2023-01-11T22:07:09.0631056Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0631372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0631646Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0632390Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.624s)
2023-01-11T22:07:09.0632414Z 
2023-01-11T22:07:09.0632673Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0632786Z Ran 1 test in 1.624s
2023-01-11T22:07:09.0632805Z 
2023-01-11T22:07:09.0632914Z OK (skipped=1)
2023-01-11T22:07:09.0632933Z 
2023-01-11T22:07:09.0633058Z Generating XML reports...
2023-01-11T22:07:09.0633489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215838.xml
2023-01-11T22:07:09.0633863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0634035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0634412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0634602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0634674Z 
2023-01-11T22:07:09.0634788Z Running tests...
2023-01-11T22:07:09.0635051Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0635366Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0635639Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0635842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34438
2023-01-11T22:07:09.0636106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34439
2023-01-11T22:07:09.0636488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0636663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0637039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0637232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0637597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0637770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0638124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0638319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0638563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0638809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0639212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0639834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0640069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0640297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0640401Z ok (4.253s)
2023-01-11T22:07:09.0640420Z 
2023-01-11T22:07:09.0640669Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0640787Z Ran 1 test in 4.253s
2023-01-11T22:07:09.0640807Z 
2023-01-11T22:07:09.0640899Z OK
2023-01-11T22:07:09.0640918Z 
2023-01-11T22:07:09.0641042Z Generating XML reports...
2023-01-11T22:07:09.0641494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215842.xml
2023-01-11T22:07:09.0641865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0642044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0642426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0642618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0642637Z 
2023-01-11T22:07:09.0642728Z Running tests...
2023-01-11T22:07:09.0642989Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0643306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0643557Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0643575Z 
2023-01-11T22:07:09.0643832Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0643943Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0644037Z 
2023-01-11T22:07:09.0644150Z OK (skipped=1)
2023-01-11T22:07:09.0644169Z 
2023-01-11T22:07:09.0644294Z Generating XML reports...
2023-01-11T22:07:09.0644730Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215849.xml
2023-01-11T22:07:09.0645098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0645274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0645708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0645905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0645924Z 
2023-01-11T22:07:09.0646033Z Running tests...
2023-01-11T22:07:09.0646296Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0646608Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0646873Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0646892Z 
2023-01-11T22:07:09.0647134Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0647243Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0647263Z 
2023-01-11T22:07:09.0647372Z OK (skipped=1)
2023-01-11T22:07:09.0647391Z 
2023-01-11T22:07:09.0647514Z Generating XML reports...
2023-01-11T22:07:09.0647966Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215851.xml
2023-01-11T22:07:09.0648340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0648513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0648936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0649137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0649157Z 
2023-01-11T22:07:09.0649249Z Running tests...
2023-01-11T22:07:09.0649511Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0649824Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0650072Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0650292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34609
2023-01-11T22:07:09.0650509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34610
2023-01-11T22:07:09.0650883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0651057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0651423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0651610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0651973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0652147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0652525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0652716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0652962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0653208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0653677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0654058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0654289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0654518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0654897Z STAGE:2023-01-11 21:58:59 34609:34609 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0655237Z STAGE:2023-01-11 21:58:59 34610:34610 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0655568Z STAGE:2023-01-11 21:59:01 34609:34609 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0655915Z STAGE:2023-01-11 21:59:01 34609:34609 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0656252Z STAGE:2023-01-11 21:59:01 34610:34610 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0656598Z STAGE:2023-01-11 21:59:01 34610:34610 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0656908Z STAGE:2023-01-11 21:59:01 34609:34609 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0657227Z STAGE:2023-01-11 21:59:01 34610:34610 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0657558Z STAGE:2023-01-11 21:59:01 34610:34610 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0657906Z STAGE:2023-01-11 21:59:01 34610:34610 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0658233Z STAGE:2023-01-11 21:59:01 34609:34609 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0658578Z STAGE:2023-01-11 21:59:01 34609:34609 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0658687Z ok (7.863s)
2023-01-11T22:07:09.0658707Z 
2023-01-11T22:07:09.0658967Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0659060Z Ran 1 test in 7.863s
2023-01-11T22:07:09.0659098Z 
2023-01-11T22:07:09.0659172Z OK
2023-01-11T22:07:09.0659191Z 
2023-01-11T22:07:09.0659316Z Generating XML reports...
2023-01-11T22:07:09.0659764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215854.xml
2023-01-11T22:07:09.0660140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0660317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0660697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0660889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0660912Z 
2023-01-11T22:07:09.0661020Z Running tests...
2023-01-11T22:07:09.0661263Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0661573Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0661838Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0661857Z 
2023-01-11T22:07:09.0662117Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0662230Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0662249Z 
2023-01-11T22:07:09.0662357Z OK (skipped=1)
2023-01-11T22:07:09.0662376Z 
2023-01-11T22:07:09.0662499Z Generating XML reports...
2023-01-11T22:07:09.0662945Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215904.xml
2023-01-11T22:07:09.0663375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0663533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0663909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0664099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0664118Z 
2023-01-11T22:07:09.0664227Z Running tests...
2023-01-11T22:07:09.0664567Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0664890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0665150Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0665171Z 
2023-01-11T22:07:09.0665430Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0665528Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0665565Z 
2023-01-11T22:07:09.0665655Z OK (skipped=1)
2023-01-11T22:07:09.0665673Z 
2023-01-11T22:07:09.0665796Z Generating XML reports...
2023-01-11T22:07:09.0666240Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215907.xml
2023-01-11T22:07:09.0666607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0666784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0667165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0667356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0667375Z 
2023-01-11T22:07:09.0667483Z Running tests...
2023-01-11T22:07:09.0667723Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0668040Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0668295Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0668517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34798
2023-01-11T22:07:09.0668733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34799
2023-01-11T22:07:09.0669107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0669282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0669660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0669831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0670196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0670373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0670744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0670933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0671180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0671426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0671827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0672226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0672437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0672724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0672826Z ok (7.150s)
2023-01-11T22:07:09.0672846Z 
2023-01-11T22:07:09.0673108Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0673221Z Ran 1 test in 7.150s
2023-01-11T22:07:09.0673240Z 
2023-01-11T22:07:09.0673333Z OK
2023-01-11T22:07:09.0673352Z 
2023-01-11T22:07:09.0673477Z Generating XML reports...
2023-01-11T22:07:09.0673972Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215909.xml
2023-01-11T22:07:09.0674352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0674510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0674887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0675080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0675100Z 
2023-01-11T22:07:09.0675210Z Running tests...
2023-01-11T22:07:09.0675471Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0675783Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0676056Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0676809Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.610s)
2023-01-11T22:07:09.0676829Z 
2023-01-11T22:07:09.0677094Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0677192Z Ran 1 test in 1.610s
2023-01-11T22:07:09.0677232Z 
2023-01-11T22:07:09.0677323Z OK (skipped=1)
2023-01-11T22:07:09.0677342Z 
2023-01-11T22:07:09.0677466Z Generating XML reports...
2023-01-11T22:07:09.0677915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215919.xml
2023-01-11T22:07:09.0678287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0678467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0678844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0679032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0679051Z 
2023-01-11T22:07:09.0679161Z Running tests...
2023-01-11T22:07:09.0679655Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0679988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0680234Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0680454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34950
2023-01-11T22:07:09.0680673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34951
2023-01-11T22:07:09.0681050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0681225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0681607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0681780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0682239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0682411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0682781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0682970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0683286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0683540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0683942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0684337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0684553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0684800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0685024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0685261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0685663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0686048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0686151Z ok (4.311s)
2023-01-11T22:07:09.0686171Z 
2023-01-11T22:07:09.0686435Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0686550Z Ran 1 test in 4.311s
2023-01-11T22:07:09.0686570Z 
2023-01-11T22:07:09.0686644Z OK
2023-01-11T22:07:09.0686663Z 
2023-01-11T22:07:09.0686788Z Generating XML reports...
2023-01-11T22:07:09.0687242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215923.xml
2023-01-11T22:07:09.0687615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0687792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0688175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0688365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0688384Z 
2023-01-11T22:07:09.0688494Z Running tests...
2023-01-11T22:07:09.0688737Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0689056Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0689303Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0689520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35057
2023-01-11T22:07:09.0689737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35058
2023-01-11T22:07:09.0690102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0690282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0690661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0690848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0691195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0691428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0691802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0691988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0692233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0692522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0692931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0693328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0693560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0693777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0693881Z ok (6.863s)
2023-01-11T22:07:09.0693901Z 
2023-01-11T22:07:09.0694166Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0694278Z Ran 1 test in 6.863s
2023-01-11T22:07:09.0694297Z 
2023-01-11T22:07:09.0694387Z OK
2023-01-11T22:07:09.0694406Z 
2023-01-11T22:07:09.0694531Z Generating XML reports...
2023-01-11T22:07:09.0694984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215930.xml
2023-01-11T22:07:09.0695358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0695516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0695896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0696092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0696111Z 
2023-01-11T22:07:09.0696220Z Running tests...
2023-01-11T22:07:09.0696484Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0696795Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0697039Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0697261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35172
2023-01-11T22:07:09.0697463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35173
2023-01-11T22:07:09.0697833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0698008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0698392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0698581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0698944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0699115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0699489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0699679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0699907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0700145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0700604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0700997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0701228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0701450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0701554Z ok (4.415s)
2023-01-11T22:07:09.0701574Z 
2023-01-11T22:07:09.0701879Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0701999Z Ran 1 test in 4.415s
2023-01-11T22:07:09.0702019Z 
2023-01-11T22:07:09.0702094Z OK
2023-01-11T22:07:09.0702112Z 
2023-01-11T22:07:09.0702238Z Generating XML reports...
2023-01-11T22:07:09.0702691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215939.xml
2023-01-11T22:07:09.0703066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0703241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0703620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0703814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0703834Z 
2023-01-11T22:07:09.0703940Z Running tests...
2023-01-11T22:07:09.0704185Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0704501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0704764Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0704987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35277
2023-01-11T22:07:09.0705209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35278
2023-01-11T22:07:09.0705580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0705754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0706125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0706317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0706663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0706832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0707204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0707396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0707640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0707885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0708283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0708680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0708911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0709135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0709360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0709657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0710056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0710445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0710546Z ok (4.336s)
2023-01-11T22:07:09.0710565Z 
2023-01-11T22:07:09.0710870Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0710988Z Ran 1 test in 4.336s
2023-01-11T22:07:09.0711008Z 
2023-01-11T22:07:09.0711082Z OK
2023-01-11T22:07:09.0711120Z 
2023-01-11T22:07:09.0711226Z Generating XML reports...
2023-01-11T22:07:09.0711679Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215946.xml
2023-01-11T22:07:09.0712047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0712227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0712609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0712799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0712818Z 
2023-01-11T22:07:09.0712926Z Running tests...
2023-01-11T22:07:09.0713184Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0713481Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0713740Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0713960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35384
2023-01-11T22:07:09.0714178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35385
2023-01-11T22:07:09.0714550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0714722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0715106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0715298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0715647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0715820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0716195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0716383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0716633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0716876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0717271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0717666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0717900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0718125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0718348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0718583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0719043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0719646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0719758Z ok (4.345s)
2023-01-11T22:07:09.0719779Z 
2023-01-11T22:07:09.0720044Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0720159Z Ran 1 test in 4.346s
2023-01-11T22:07:09.0720179Z 
2023-01-11T22:07:09.0720344Z OK
2023-01-11T22:07:09.0720367Z 
2023-01-11T22:07:09.0720481Z Generating XML reports...
2023-01-11T22:07:09.0720935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215953.xml
2023-01-11T22:07:09.0721304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0721483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0721864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0722056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0722075Z 
2023-01-11T22:07:09.0722182Z Running tests...
2023-01-11T22:07:09.0722443Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0722752Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0723013Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0723233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35491
2023-01-11T22:07:09.0723448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35492
2023-01-11T22:07:09.0723823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0724000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0724380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0724573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0724937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0725097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0725475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0725663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0725909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0726159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0726555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0726948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0727180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0727411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0727496Z ok (6.378s)
2023-01-11T22:07:09.0727516Z 
2023-01-11T22:07:09.0727785Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0727898Z Ran 1 test in 6.378s
2023-01-11T22:07:09.0727917Z 
2023-01-11T22:07:09.0728007Z OK
2023-01-11T22:07:09.0728092Z 
2023-01-11T22:07:09.0728224Z Generating XML reports...
2023-01-11T22:07:09.0728677Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220000.xml
2023-01-11T22:07:09.0729045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0729220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0729624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0729820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0729841Z 
2023-01-11T22:07:09.0729951Z Running tests...
2023-01-11T22:07:09.0730212Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0730521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0730762Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2023-01-11T22:07:09.0730782Z 
2023-01-11T22:07:09.0731037Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0731149Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0731168Z 
2023-01-11T22:07:09.0731274Z OK (skipped=1)
2023-01-11T22:07:09.0731293Z 
2023-01-11T22:07:09.0731398Z Generating XML reports...
2023-01-11T22:07:09.0731845Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220009.xml
2023-01-11T22:07:09.0732221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0732393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0732772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0732967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0732986Z 
2023-01-11T22:07:09.0733094Z Running tests...
2023-01-11T22:07:09.0733353Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0733650Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0733888Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T22:07:09.0733907Z 
2023-01-11T22:07:09.0734166Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0734278Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0734298Z 
2023-01-11T22:07:09.0734406Z OK (skipped=1)
2023-01-11T22:07:09.0734425Z 
2023-01-11T22:07:09.0734548Z Generating XML reports...
2023-01-11T22:07:09.0734994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220011.xml
2023-01-11T22:07:09.0735365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0735545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0735905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0736096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0736115Z 
2023-01-11T22:07:09.0736224Z Running tests...
2023-01-11T22:07:09.0736484Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0736796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0737059Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T22:07:09.0737079Z 
2023-01-11T22:07:09.0737337Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0737509Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0737528Z 
2023-01-11T22:07:09.0737638Z OK (skipped=1)
2023-01-11T22:07:09.0737657Z 
2023-01-11T22:07:09.0737762Z Generating XML reports...
2023-01-11T22:07:09.0738213Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220014.xml
2023-01-11T22:07:09.0738580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0738754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0739177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0739373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0739393Z 
2023-01-11T22:07:09.0739502Z Running tests...
2023-01-11T22:07:09.0739765Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0740064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0740326Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T22:07:09.0740346Z 
2023-01-11T22:07:09.0740604Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0740715Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0740735Z 
2023-01-11T22:07:09.0740842Z OK (skipped=1)
2023-01-11T22:07:09.0740861Z 
2023-01-11T22:07:09.0740984Z Generating XML reports...
2023-01-11T22:07:09.0741430Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220016.xml
2023-01-11T22:07:09.0741795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0741972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0742336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0742528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0742548Z 
2023-01-11T22:07:09.0742656Z Running tests...
2023-01-11T22:07:09.0742914Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0743225Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0743511Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0743730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35741
2023-01-11T22:07:09.0743946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35742
2023-01-11T22:07:09.0744314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0744476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0744854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0745045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0745406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0745580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0745956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0746143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0746391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0746616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0747077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0747473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0747702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0747997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0748229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0748465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0748904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0749299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0749537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:09.0749761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:09.0750157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0750550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0750784Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:07:09.0750888Z ok (22.482s)
2023-01-11T22:07:09.0750909Z 
2023-01-11T22:07:09.0751175Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0751295Z Ran 1 test in 22.482s
2023-01-11T22:07:09.0751314Z 
2023-01-11T22:07:09.0751409Z OK
2023-01-11T22:07:09.0751428Z 
2023-01-11T22:07:09.0751533Z Generating XML reports...
2023-01-11T22:07:09.0751982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220018.xml
2023-01-11T22:07:09.0752354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0752527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0752908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0753099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0753118Z 
2023-01-11T22:07:09.0753233Z Running tests...
2023-01-11T22:07:09.0753492Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0753811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0754096Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0754316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35864
2023-01-11T22:07:09.0754535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35865
2023-01-11T22:07:09.0754909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0755085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0755463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0755655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0756018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0756232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0756608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0756793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0757038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0757327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0757735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0758129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0758362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0758597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.0758807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0759049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.0759668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0760080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.0760321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:09.0760559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:09.0760956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0761348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.0761583Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:07:09.0761793Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:07:09.0761897Z ok (22.473s)
2023-01-11T22:07:09.0761921Z 
2023-01-11T22:07:09.0762190Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0762305Z Ran 1 test in 22.473s
2023-01-11T22:07:09.0762324Z 
2023-01-11T22:07:09.0762418Z OK
2023-01-11T22:07:09.0762437Z 
2023-01-11T22:07:09.0762562Z Generating XML reports...
2023-01-11T22:07:09.0763011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220043.xml
2023-01-11T22:07:09.0763387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0763565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0763925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0764117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0764136Z 
2023-01-11T22:07:09.0764249Z Running tests...
2023-01-11T22:07:09.0764509Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0764820Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0765230Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:09.0765325Z 
2023-01-11T22:07:09.0765593Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0765706Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0765726Z 
2023-01-11T22:07:09.0765834Z OK (skipped=1)
2023-01-11T22:07:09.0765854Z 
2023-01-11T22:07:09.0765959Z Generating XML reports...
2023-01-11T22:07:09.0766406Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220109.xml
2023-01-11T22:07:09.0766834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0767017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0767397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0767586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0767605Z 
2023-01-11T22:07:09.0767718Z Running tests...
2023-01-11T22:07:09.0767978Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0768272Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0768668Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:09.0768688Z 
2023-01-11T22:07:09.0768947Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0769058Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0769077Z 
2023-01-11T22:07:09.0769189Z OK (skipped=1)
2023-01-11T22:07:09.0769208Z 
2023-01-11T22:07:09.0769328Z Generating XML reports...
2023-01-11T22:07:09.0769772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220111.xml
2023-01-11T22:07:09.0770143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0770321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0770682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0770871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0770891Z 
2023-01-11T22:07:09.0770998Z Running tests...
2023-01-11T22:07:09.0771259Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0771575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0771999Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:09.0772019Z 
2023-01-11T22:07:09.0772276Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0772388Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0772411Z 
2023-01-11T22:07:09.0772521Z OK (skipped=1)
2023-01-11T22:07:09.0772540Z 
2023-01-11T22:07:09.0772646Z Generating XML reports...
2023-01-11T22:07:09.0773093Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220113.xml
2023-01-11T22:07:09.0773462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0773635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0774016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0774206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0774226Z 
2023-01-11T22:07:09.0774333Z Running tests...
2023-01-11T22:07:09.0774591Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0774902Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0775359Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:09.0775380Z 
2023-01-11T22:07:09.0775636Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0775747Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0775766Z 
2023-01-11T22:07:09.0775873Z OK (skipped=1)
2023-01-11T22:07:09.0775892Z 
2023-01-11T22:07:09.0776016Z Generating XML reports...
2023-01-11T22:07:09.0776512Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220116.xml
2023-01-11T22:07:09.0776889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0777062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0777439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0777618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0777637Z 
2023-01-11T22:07:09.0777746Z Running tests...
2023-01-11T22:07:09.0778006Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0778319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0778732Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:09.0778752Z 
2023-01-11T22:07:09.0779009Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0779119Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0779138Z 
2023-01-11T22:07:09.0779243Z OK (skipped=1)
2023-01-11T22:07:09.0779262Z 
2023-01-11T22:07:09.0779367Z Generating XML reports...
2023-01-11T22:07:09.0779813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220118.xml
2023-01-11T22:07:09.0780187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0780361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0780738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0780930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0780952Z 
2023-01-11T22:07:09.0781064Z Running tests...
2023-01-11T22:07:09.0781324Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0781636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0781892Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0782116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36152
2023-01-11T22:07:09.0782330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36153
2023-01-11T22:07:09.0782694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0782869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0783251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0783441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0783801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0783957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0784333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0784583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0784828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0785075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0785478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0785916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0786154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0786384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0786468Z ok (5.850s)
2023-01-11T22:07:09.0786509Z 
2023-01-11T22:07:09.0786759Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0786871Z Ran 1 test in 5.851s
2023-01-11T22:07:09.0786890Z 
2023-01-11T22:07:09.0786984Z OK
2023-01-11T22:07:09.0787003Z 
2023-01-11T22:07:09.0787129Z Generating XML reports...
2023-01-11T22:07:09.0787579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220120.xml
2023-01-11T22:07:09.0787956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0788131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0788513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0788686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0788705Z 
2023-01-11T22:07:09.0788819Z Running tests...
2023-01-11T22:07:09.0789076Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0789388Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0789659Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0789878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36266
2023-01-11T22:07:09.0790101Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36267
2023-01-11T22:07:09.0790473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0790629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0791007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0791201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0791567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0791737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0792109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0792296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0792542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0792786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0793166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0793564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0793851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0794080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0794182Z ok (5.853s)
2023-01-11T22:07:09.0794202Z 
2023-01-11T22:07:09.0794470Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0794582Z Ran 1 test in 5.853s
2023-01-11T22:07:09.0794601Z 
2023-01-11T22:07:09.0794738Z OK
2023-01-11T22:07:09.0794759Z 
2023-01-11T22:07:09.0794869Z Generating XML reports...
2023-01-11T22:07:09.0795320Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220129.xml
2023-01-11T22:07:09.0795685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0795860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0796239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0796429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0796449Z 
2023-01-11T22:07:09.0796558Z Running tests...
2023-01-11T22:07:09.0796818Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0797136Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0797393Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0797613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36380
2023-01-11T22:07:09.0797829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36381
2023-01-11T22:07:09.0798200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0798378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0798745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0798917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0799298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0799683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0800078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0800268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0800514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0800763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0801166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0801566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0801797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0802023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0802108Z ok (6.951s)
2023-01-11T22:07:09.0802146Z 
2023-01-11T22:07:09.0802393Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0802506Z Ran 1 test in 6.952s
2023-01-11T22:07:09.0802526Z 
2023-01-11T22:07:09.0802618Z OK
2023-01-11T22:07:09.0802713Z 
2023-01-11T22:07:09.0802844Z Generating XML reports...
2023-01-11T22:07:09.0803298Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220137.xml
2023-01-11T22:07:09.0803666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0803844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0804278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0804460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0804479Z 
2023-01-11T22:07:09.0804588Z Running tests...
2023-01-11T22:07:09.0804855Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0805167Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0805438Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0805658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36495
2023-01-11T22:07:09.0805876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36496
2023-01-11T22:07:09.0806247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0806405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0806781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0806969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0807330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0807503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0807881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0808069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0808316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0808562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0808947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0809347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0809578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0809804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0809911Z ok (5.954s)
2023-01-11T22:07:09.0809931Z 
2023-01-11T22:07:09.0810194Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0810308Z Ran 1 test in 5.954s
2023-01-11T22:07:09.0810328Z 
2023-01-11T22:07:09.0810420Z OK
2023-01-11T22:07:09.0810438Z 
2023-01-11T22:07:09.0810545Z Generating XML reports...
2023-01-11T22:07:09.0810991Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220147.xml
2023-01-11T22:07:09.0811365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0811541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0811918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0812108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0812210Z 
2023-01-11T22:07:09.0812325Z Running tests...
2023-01-11T22:07:09.0812587Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0812900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0813157Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0813968Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2023-01-11T22:07:09.0813991Z 
2023-01-11T22:07:09.0814257Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0814369Z Ran 1 test in 1.640s
2023-01-11T22:07:09.0814391Z 
2023-01-11T22:07:09.0814498Z OK (skipped=1)
2023-01-11T22:07:09.0814517Z 
2023-01-11T22:07:09.0814637Z Generating XML reports...
2023-01-11T22:07:09.0815090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220155.xml
2023-01-11T22:07:09.0815459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0815634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0816016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0816187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0816206Z 
2023-01-11T22:07:09.0816315Z Running tests...
2023-01-11T22:07:09.0816579Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0816892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0817146Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:07:09.0817166Z 
2023-01-11T22:07:09.0817425Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0817539Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0817558Z 
2023-01-11T22:07:09.0817667Z OK (skipped=1)
2023-01-11T22:07:09.0817686Z 
2023-01-11T22:07:09.0817791Z Generating XML reports...
2023-01-11T22:07:09.0818241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220159.xml
2023-01-11T22:07:09.0818609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0818784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0819162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0819352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0819371Z 
2023-01-11T22:07:09.0819479Z Running tests...
2023-01-11T22:07:09.0819741Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0820054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0820309Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:07:09.0820347Z 
2023-01-11T22:07:09.0820589Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0820701Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0820720Z 
2023-01-11T22:07:09.0820825Z OK (skipped=1)
2023-01-11T22:07:09.0820844Z 
2023-01-11T22:07:09.0820969Z Generating XML reports...
2023-01-11T22:07:09.0821415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220202.xml
2023-01-11T22:07:09.0821846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0822022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0822400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0822572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0822591Z 
2023-01-11T22:07:09.0822744Z Running tests...
2023-01-11T22:07:09.0823011Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0823321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0823633Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:07:09.0823657Z 
2023-01-11T22:07:09.0823919Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0824029Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0824048Z 
2023-01-11T22:07:09.0824158Z OK (skipped=1)
2023-01-11T22:07:09.0824177Z 
2023-01-11T22:07:09.0824301Z Generating XML reports...
2023-01-11T22:07:09.0824729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220204.xml
2023-01-11T22:07:09.0825099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0825273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0825653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0825842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0825866Z 
2023-01-11T22:07:09.0825974Z Running tests...
2023-01-11T22:07:09.0826232Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0826546Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0826832Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0827053Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36742
2023-01-11T22:07:09.0827270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36743
2023-01-11T22:07:09.0827644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0827817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0828194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0828388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0828750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0828922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0829274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0829463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0829711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0829959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0830360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0830817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0831050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0831279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0831379Z ok (4.254s)
2023-01-11T22:07:09.0831398Z 
2023-01-11T22:07:09.0831644Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0831758Z Ran 1 test in 4.255s
2023-01-11T22:07:09.0831823Z 
2023-01-11T22:07:09.0831921Z OK
2023-01-11T22:07:09.0831940Z 
2023-01-11T22:07:09.0832065Z Generating XML reports...
2023-01-11T22:07:09.0832516Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220207.xml
2023-01-11T22:07:09.0832883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0833062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0833441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0833614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0833651Z 
2023-01-11T22:07:09.0833743Z Running tests...
2023-01-11T22:07:09.0834004Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0834319Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0834611Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0834831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36847
2023-01-11T22:07:09.0835046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36848
2023-01-11T22:07:09.0835417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0835592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0835951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0836143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0836506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0836678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0837054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0837293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0837547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0837790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0838176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0838575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0838810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0839040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0839143Z ok (4.326s)
2023-01-11T22:07:09.0839163Z 
2023-01-11T22:07:09.0839622Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0839743Z Ran 1 test in 4.326s
2023-01-11T22:07:09.0839840Z 
2023-01-11T22:07:09.0839937Z OK
2023-01-11T22:07:09.0839957Z 
2023-01-11T22:07:09.0840083Z Generating XML reports...
2023-01-11T22:07:09.0840526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220213.xml
2023-01-11T22:07:09.0840895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0841070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0841505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0841704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0841724Z 
2023-01-11T22:07:09.0841831Z Running tests...
2023-01-11T22:07:09.0842095Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0842406Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0842672Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:07:09.0842710Z 
2023-01-11T22:07:09.0842950Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0843063Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0843082Z 
2023-01-11T22:07:09.0843195Z OK (skipped=1)
2023-01-11T22:07:09.0843213Z 
2023-01-11T22:07:09.0843333Z Generating XML reports...
2023-01-11T22:07:09.0843782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220220.xml
2023-01-11T22:07:09.0844152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0844329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0844706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0844884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0844922Z 
2023-01-11T22:07:09.0845012Z Running tests...
2023-01-11T22:07:09.0845271Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0845583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0845887Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:07:09.0845907Z 
2023-01-11T22:07:09.0846165Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0846278Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0846297Z 
2023-01-11T22:07:09.0846406Z OK (skipped=1)
2023-01-11T22:07:09.0846425Z 
2023-01-11T22:07:09.0846546Z Generating XML reports...
2023-01-11T22:07:09.0846973Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220223.xml
2023-01-11T22:07:09.0847350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0847523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0847901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0848091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0848114Z 
2023-01-11T22:07:09.0848223Z Running tests...
2023-01-11T22:07:09.0848485Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0848840Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0849120Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0849924Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.652s)
2023-01-11T22:07:09.0849962Z 
2023-01-11T22:07:09.0850203Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0850315Z Ran 1 test in 1.652s
2023-01-11T22:07:09.0850334Z 
2023-01-11T22:07:09.0850442Z OK (skipped=1)
2023-01-11T22:07:09.0850506Z 
2023-01-11T22:07:09.0850635Z Generating XML reports...
2023-01-11T22:07:09.0851079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220225.xml
2023-01-11T22:07:09.0851448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0851624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0852008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0852199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0852219Z 
2023-01-11T22:07:09.0852310Z Running tests...
2023-01-11T22:07:09.0852569Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0852880Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0853165Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0853385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37052
2023-01-11T22:07:09.0853600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37053
2023-01-11T22:07:09.0853977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0854154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0854517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0854708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0855074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0855249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0855619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0855809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0856052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0856301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0856703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0857081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0857314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0857545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0857649Z ok (7.375s)
2023-01-11T22:07:09.0857669Z 
2023-01-11T22:07:09.0857931Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0858042Z Ran 1 test in 7.375s
2023-01-11T22:07:09.0858061Z 
2023-01-11T22:07:09.0858155Z OK
2023-01-11T22:07:09.0858174Z 
2023-01-11T22:07:09.0858358Z Generating XML reports...
2023-01-11T22:07:09.0858795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220229.xml
2023-01-11T22:07:09.0859166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0859342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0859719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0859952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0859973Z 
2023-01-11T22:07:09.0860086Z Running tests...
2023-01-11T22:07:09.0860347Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0860660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0860933Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0861141Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37171
2023-01-11T22:07:09.0861358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37172
2023-01-11T22:07:09.0861725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0861898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0862273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0862464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0862826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0863002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0863362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0863548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0863796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0864039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0864444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0864839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0865070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0865297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0865402Z ok (6.940s)
2023-01-11T22:07:09.0865422Z 
2023-01-11T22:07:09.0865671Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0865785Z Ran 1 test in 6.940s
2023-01-11T22:07:09.0865804Z 
2023-01-11T22:07:09.0865898Z OK
2023-01-11T22:07:09.0865916Z 
2023-01-11T22:07:09.0866042Z Generating XML reports...
2023-01-11T22:07:09.0866492Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220239.xml
2023-01-11T22:07:09.0866862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0867039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0867420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0867608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0867683Z 
2023-01-11T22:07:09.0867779Z Running tests...
2023-01-11T22:07:09.0868044Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0868359Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0868649Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0868868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37286
2023-01-11T22:07:09.0869128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37287
2023-01-11T22:07:09.0869511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0869687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0870045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0870238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0870602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0870775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0871146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0871336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0871580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0871823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0872224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0872603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0872832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0873058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0873162Z ok (7.045s)
2023-01-11T22:07:09.0873182Z 
2023-01-11T22:07:09.0873445Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0873558Z Ran 1 test in 7.045s
2023-01-11T22:07:09.0873578Z 
2023-01-11T22:07:09.0873671Z OK
2023-01-11T22:07:09.0873691Z 
2023-01-11T22:07:09.0873816Z Generating XML reports...
2023-01-11T22:07:09.0874246Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220249.xml
2023-01-11T22:07:09.0874618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0874799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0875177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0875371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0875390Z 
2023-01-11T22:07:09.0875499Z Running tests...
2023-01-11T22:07:09.0875754Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0876071Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0876346Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0877077Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T22:07:09.0877167Z 
2023-01-11T22:07:09.0877416Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0877531Z Ran 1 test in 1.634s
2023-01-11T22:07:09.0877550Z 
2023-01-11T22:07:09.0877660Z OK (skipped=1)
2023-01-11T22:07:09.0877679Z 
2023-01-11T22:07:09.0877801Z Generating XML reports...
2023-01-11T22:07:09.0878302Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220258.xml
2023-01-11T22:07:09.0878681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0878861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0879238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0879645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0879667Z 
2023-01-11T22:07:09.0879766Z Running tests...
2023-01-11T22:07:09.0880035Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0880351Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0880646Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0881400Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2023-01-11T22:07:09.0881421Z 
2023-01-11T22:07:09.0881679Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0881795Z Ran 1 test in 1.628s
2023-01-11T22:07:09.0881814Z 
2023-01-11T22:07:09.0881922Z OK (skipped=1)
2023-01-11T22:07:09.0881940Z 
2023-01-11T22:07:09.0882065Z Generating XML reports...
2023-01-11T22:07:09.0882494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220303.xml
2023-01-11T22:07:09.0882863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0883042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0883425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0883617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0883636Z 
2023-01-11T22:07:09.0883744Z Running tests...
2023-01-11T22:07:09.0884003Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0884323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0884635Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0884840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37469
2023-01-11T22:07:09.0885060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37470
2023-01-11T22:07:09.0885433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0885609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0885987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0886177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0886629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0886801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0887155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0887346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0887649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0887905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0888308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0888703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0888936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0889164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0889314Z skip: Need at least 4 CUDA devices (4.341s)
2023-01-11T22:07:09.0889335Z 
2023-01-11T22:07:09.0889580Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0889694Z Ran 1 test in 4.341s
2023-01-11T22:07:09.0889713Z 
2023-01-11T22:07:09.0889820Z OK (skipped=1)
2023-01-11T22:07:09.0889841Z 
2023-01-11T22:07:09.0889966Z Generating XML reports...
2023-01-11T22:07:09.0890419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220307.xml
2023-01-11T22:07:09.0890787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0890961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0891343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0891532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0891552Z 
2023-01-11T22:07:09.0891642Z Running tests...
2023-01-11T22:07:09.0891903Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0892213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0892538Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0892754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37574
2023-01-11T22:07:09.0892973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37575
2023-01-11T22:07:09.0893346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0893522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0893883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0894073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0894435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0894615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0894991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0895178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0895422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0895725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0896125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0896503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0896732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0897004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0897162Z skip: Need at least 4 CUDA devices (4.247s)
2023-01-11T22:07:09.0897183Z 
2023-01-11T22:07:09.0897449Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0897561Z Ran 1 test in 4.248s
2023-01-11T22:07:09.0897580Z 
2023-01-11T22:07:09.0897691Z OK (skipped=1)
2023-01-11T22:07:09.0897710Z 
2023-01-11T22:07:09.0897835Z Generating XML reports...
2023-01-11T22:07:09.0898287Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220314.xml
2023-01-11T22:07:09.0898640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0898816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0899197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0899388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0899407Z 
2023-01-11T22:07:09.0899519Z Running tests...
2023-01-11T22:07:09.0899783Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0900096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0900381Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0901132Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2023-01-11T22:07:09.0901153Z 
2023-01-11T22:07:09.0901419Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0901515Z Ran 1 test in 1.638s
2023-01-11T22:07:09.0901534Z 
2023-01-11T22:07:09.0901643Z OK (skipped=1)
2023-01-11T22:07:09.0901662Z 
2023-01-11T22:07:09.0901786Z Generating XML reports...
2023-01-11T22:07:09.0902233Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220320.xml
2023-01-11T22:07:09.0902607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0902782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0903160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0903351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0903371Z 
2023-01-11T22:07:09.0903462Z Running tests...
2023-01-11T22:07:09.0903728Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0904041Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0904310Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0904330Z 
2023-01-11T22:07:09.0904589Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0904756Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0904776Z 
2023-01-11T22:07:09.0904882Z OK (skipped=1)
2023-01-11T22:07:09.0904901Z 
2023-01-11T22:07:09.0905024Z Generating XML reports...
2023-01-11T22:07:09.0905470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220325.xml
2023-01-11T22:07:09.0905820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0906037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0906423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0906612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0906631Z 
2023-01-11T22:07:09.0906739Z Running tests...
2023-01-11T22:07:09.0906997Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0907313Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0907582Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0907601Z 
2023-01-11T22:07:09.0907857Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0907951Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0907970Z 
2023-01-11T22:07:09.0908074Z OK (skipped=1)
2023-01-11T22:07:09.0908093Z 
2023-01-11T22:07:09.0908219Z Generating XML reports...
2023-01-11T22:07:09.0908665Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220327.xml
2023-01-11T22:07:09.0909039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0909213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0909597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0909789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0909809Z 
2023-01-11T22:07:09.0909901Z Running tests...
2023-01-11T22:07:09.0910160Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0910474Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0910754Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0910773Z 
2023-01-11T22:07:09.0911033Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0911144Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0911163Z 
2023-01-11T22:07:09.0911273Z OK (skipped=1)
2023-01-11T22:07:09.0911292Z 
2023-01-11T22:07:09.0911414Z Generating XML reports...
2023-01-11T22:07:09.0911860Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220329.xml
2023-01-11T22:07:09.0912212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0912389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0912761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0912956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0912975Z 
2023-01-11T22:07:09.0913081Z Running tests...
2023-01-11T22:07:09.0913345Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0913658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0913925Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0913996Z 
2023-01-11T22:07:09.0914261Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0914357Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0914376Z 
2023-01-11T22:07:09.0914481Z OK (skipped=1)
2023-01-11T22:07:09.0914500Z 
2023-01-11T22:07:09.0914623Z Generating XML reports...
2023-01-11T22:07:09.0915064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220332.xml
2023-01-11T22:07:09.0915475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0915656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0916033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0916226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0916249Z 
2023-01-11T22:07:09.0916357Z Running tests...
2023-01-11T22:07:09.0916601Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0916912Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0917171Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0917191Z 
2023-01-11T22:07:09.0917447Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0917557Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0917576Z 
2023-01-11T22:07:09.0917681Z OK (skipped=1)
2023-01-11T22:07:09.0917700Z 
2023-01-11T22:07:09.0917821Z Generating XML reports...
2023-01-11T22:07:09.0918264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220334.xml
2023-01-11T22:07:09.0918613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0918792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0919174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0919365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0919560Z 
2023-01-11T22:07:09.0919677Z Running tests...
2023-01-11T22:07:09.0919950Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0920266Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0920534Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0920553Z 
2023-01-11T22:07:09.0920812Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0920911Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0920930Z 
2023-01-11T22:07:09.0921038Z OK (skipped=1)
2023-01-11T22:07:09.0921057Z 
2023-01-11T22:07:09.0921179Z Generating XML reports...
2023-01-11T22:07:09.0921628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220337.xml
2023-01-11T22:07:09.0922000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0922176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0922559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0922747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0922766Z 
2023-01-11T22:07:09.0922875Z Running tests...
2023-01-11T22:07:09.0923118Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0923515Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0923783Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0923803Z 
2023-01-11T22:07:09.0924059Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0924171Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0924190Z 
2023-01-11T22:07:09.0924298Z OK (skipped=1)
2023-01-11T22:07:09.0924317Z 
2023-01-11T22:07:09.0924445Z Generating XML reports...
2023-01-11T22:07:09.0924950Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220339.xml
2023-01-11T22:07:09.0925313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0925487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0925868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0926056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0926075Z 
2023-01-11T22:07:09.0926183Z Running tests...
2023-01-11T22:07:09.0926441Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0926750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0927016Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0927036Z 
2023-01-11T22:07:09.0927290Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0927383Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0927418Z 
2023-01-11T22:07:09.0927506Z OK (skipped=1)
2023-01-11T22:07:09.0927525Z 
2023-01-11T22:07:09.0927647Z Generating XML reports...
2023-01-11T22:07:09.0928098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220341.xml
2023-01-11T22:07:09.0928468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0928641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0929020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0929211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0929231Z 
2023-01-11T22:07:09.0929341Z Running tests...
2023-01-11T22:07:09.0929585Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0929896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0930148Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0930171Z 
2023-01-11T22:07:09.0930430Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0930542Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0930561Z 
2023-01-11T22:07:09.0930666Z OK (skipped=1)
2023-01-11T22:07:09.0930685Z 
2023-01-11T22:07:09.0930810Z Generating XML reports...
2023-01-11T22:07:09.0931258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220344.xml
2023-01-11T22:07:09.0931633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0931792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0932172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0932364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0932437Z 
2023-01-11T22:07:09.0932550Z Running tests...
2023-01-11T22:07:09.0932814Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0933125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0933378Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0933398Z 
2023-01-11T22:07:09.0933659Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0933753Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0933833Z 
2023-01-11T22:07:09.0933930Z OK (skipped=1)
2023-01-11T22:07:09.0933949Z 
2023-01-11T22:07:09.0934072Z Generating XML reports...
2023-01-11T22:07:09.0934520Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220346.xml
2023-01-11T22:07:09.0934890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0935068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0935446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0935639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0935658Z 
2023-01-11T22:07:09.0935768Z Running tests...
2023-01-11T22:07:09.0936013Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0936329Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0936584Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0936801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38043
2023-01-11T22:07:09.0937019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38044
2023-01-11T22:07:09.0937396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0937571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0937945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0938120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0938491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0938665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0939037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0939226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0939471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0939724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0940130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0940525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0940743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0940975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0941309Z STAGE:2023-01-11 22:03:54 38043:38043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0942077Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:07:09.0942251Z   warnings.warn(
2023-01-11T22:07:09.0942592Z STAGE:2023-01-11 22:03:54 38044:38044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0943407Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:07:09.0943528Z   warnings.warn(
2023-01-11T22:07:09.0943869Z STAGE:2023-01-11 22:03:54 38043:38043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0944203Z STAGE:2023-01-11 22:03:54 38044:38044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0944533Z STAGE:2023-01-11 22:03:54 38043:38043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0944887Z STAGE:2023-01-11 22:03:54 38044:38044 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0945215Z STAGE:2023-01-11 22:03:54 38044:38044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0945548Z STAGE:2023-01-11 22:03:54 38044:38044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0945893Z STAGE:2023-01-11 22:03:54 38044:38044 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0946222Z STAGE:2023-01-11 22:03:54 38043:38043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0946555Z STAGE:2023-01-11 22:03:54 38043:38043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0946899Z STAGE:2023-01-11 22:03:54 38043:38043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0947007Z ok (6.458s)
2023-01-11T22:07:09.0947027Z 
2023-01-11T22:07:09.0947276Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0947389Z Ran 1 test in 6.458s
2023-01-11T22:07:09.0947408Z 
2023-01-11T22:07:09.0947497Z OK
2023-01-11T22:07:09.0947516Z 
2023-01-11T22:07:09.0947640Z Generating XML reports...
2023-01-11T22:07:09.0948094Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220349.xml
2023-01-11T22:07:09.0948470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0948648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0949071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0949246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0949288Z 
2023-01-11T22:07:09.0949381Z Running tests...
2023-01-11T22:07:09.0949640Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0949955Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0950214Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0950233Z 
2023-01-11T22:07:09.0950490Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0950602Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0950624Z 
2023-01-11T22:07:09.0950730Z OK (skipped=1)
2023-01-11T22:07:09.0950749Z 
2023-01-11T22:07:09.0950872Z Generating XML reports...
2023-01-11T22:07:09.0951307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220358.xml
2023-01-11T22:07:09.0951677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0951914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0952298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0952490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0952510Z 
2023-01-11T22:07:09.0952617Z Running tests...
2023-01-11T22:07:09.0952879Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0953267Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0953548Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0953752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38194
2023-01-11T22:07:09.0953971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38195
2023-01-11T22:07:09.0954350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0954521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0954890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0955067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0955450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0955642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0956002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0956191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0956439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0956689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0957092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0957492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0957728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0957957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0958293Z STAGE:2023-01-11 22:04:06 38194:38194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0958596Z STAGE:2023-01-11 22:04:06 38195:38195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0958936Z STAGE:2023-01-11 22:04:07 38194:38194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0959273Z STAGE:2023-01-11 22:04:07 38195:38195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0959843Z STAGE:2023-01-11 22:04:07 38194:38194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0960197Z STAGE:2023-01-11 22:04:07 38195:38195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0960527Z STAGE:2023-01-11 22:04:07 38195:38195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0960859Z STAGE:2023-01-11 22:04:07 38195:38195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0961205Z STAGE:2023-01-11 22:04:07 38195:38195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0961526Z STAGE:2023-01-11 22:04:07 38194:38194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0961920Z STAGE:2023-01-11 22:04:07 38194:38194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0962261Z STAGE:2023-01-11 22:04:07 38194:38194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0962365Z ok (7.959s)
2023-01-11T22:07:09.0962385Z 
2023-01-11T22:07:09.0962647Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0962759Z Ran 1 test in 7.959s
2023-01-11T22:07:09.0962778Z 
2023-01-11T22:07:09.0962873Z OK
2023-01-11T22:07:09.0962949Z 
2023-01-11T22:07:09.0963082Z Generating XML reports...
2023-01-11T22:07:09.0963536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220400.xml
2023-01-11T22:07:09.0963889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0964068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0964455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0964649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0964668Z 
2023-01-11T22:07:09.0964774Z Running tests...
2023-01-11T22:07:09.0965035Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0965352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0965622Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0965841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38313
2023-01-11T22:07:09.0966041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38314
2023-01-11T22:07:09.0966407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0966587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0966966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0967158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0967518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0967693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0968068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0968240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0968486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0968737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0969135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0969530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0969759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0969990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0970094Z ok (7.029s)
2023-01-11T22:07:09.0970113Z 
2023-01-11T22:07:09.0970386Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0970481Z Ran 1 test in 7.030s
2023-01-11T22:07:09.0970500Z 
2023-01-11T22:07:09.0970593Z OK
2023-01-11T22:07:09.0970612Z 
2023-01-11T22:07:09.0970737Z Generating XML reports...
2023-01-11T22:07:09.0971250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220410.xml
2023-01-11T22:07:09.0971621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0971798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0972175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0972540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0972562Z 
2023-01-11T22:07:09.0972681Z Running tests...
2023-01-11T22:07:09.0972929Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0973242Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0973497Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.0973521Z 
2023-01-11T22:07:09.0973778Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0973890Z Ran 1 test in 0.002s
2023-01-11T22:07:09.0973909Z 
2023-01-11T22:07:09.0974014Z OK (skipped=1)
2023-01-11T22:07:09.0974033Z 
2023-01-11T22:07:09.0974157Z Generating XML reports...
2023-01-11T22:07:09.0974609Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220420.xml
2023-01-11T22:07:09.0974964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0975141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0975517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0975708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0975731Z 
2023-01-11T22:07:09.0975839Z Running tests...
2023-01-11T22:07:09.0976107Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0976419Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0976673Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0976890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38461
2023-01-11T22:07:09.0977094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38462
2023-01-11T22:07:09.0977464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0977638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0978015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0978210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0978578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0978753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0979125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0979298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0979549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0979789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0980192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0980653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0980881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0981111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0981445Z STAGE:2023-01-11 22:04:28 38461:38461 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0981809Z STAGE:2023-01-11 22:04:28 38462:38462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0982138Z STAGE:2023-01-11 22:04:28 38462:38462 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0982464Z STAGE:2023-01-11 22:04:28 38461:38461 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0982807Z STAGE:2023-01-11 22:04:28 38462:38462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0983156Z STAGE:2023-01-11 22:04:28 38461:38461 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0983478Z STAGE:2023-01-11 22:04:28 38461:38461 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0983816Z STAGE:2023-01-11 22:04:28 38461:38461 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0984159Z STAGE:2023-01-11 22:04:28 38461:38461 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0984483Z STAGE:2023-01-11 22:04:28 38462:38462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0984813Z STAGE:2023-01-11 22:04:28 38462:38462 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0985135Z STAGE:2023-01-11 22:04:28 38462:38462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0985238Z ok (6.342s)
2023-01-11T22:07:09.0985262Z 
2023-01-11T22:07:09.0985528Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0985641Z Ran 1 test in 6.342s
2023-01-11T22:07:09.0985660Z 
2023-01-11T22:07:09.0985752Z OK
2023-01-11T22:07:09.0985771Z 
2023-01-11T22:07:09.0985894Z Generating XML reports...
2023-01-11T22:07:09.0986347Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220422.xml
2023-01-11T22:07:09.0986719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0986896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0987260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0987451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0987470Z 
2023-01-11T22:07:09.0987578Z Running tests...
2023-01-11T22:07:09.0987842Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0988156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.0988419Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.0988638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38579
2023-01-11T22:07:09.0988855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38580
2023-01-11T22:07:09.0989212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0989388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0989766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0989959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0990386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0990558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0990928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0991118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0991409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.0991644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.0992045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0992439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.0992672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.0992895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.0993232Z STAGE:2023-01-11 22:04:37 38580:38580 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0993550Z STAGE:2023-01-11 22:04:37 38579:38579 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0993885Z STAGE:2023-01-11 22:04:37 38580:38580 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0994218Z STAGE:2023-01-11 22:04:37 38579:38579 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0994545Z STAGE:2023-01-11 22:04:37 38580:38580 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0994891Z STAGE:2023-01-11 22:04:37 38579:38579 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0995221Z STAGE:2023-01-11 22:04:37 38579:38579 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0995538Z STAGE:2023-01-11 22:04:37 38580:38580 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.0995871Z STAGE:2023-01-11 22:04:37 38579:38579 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0996213Z STAGE:2023-01-11 22:04:37 38579:38579 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0996545Z STAGE:2023-01-11 22:04:37 38580:38580 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.0996887Z STAGE:2023-01-11 22:04:37 38580:38580 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.0996972Z ok (6.483s)
2023-01-11T22:07:09.0997010Z 
2023-01-11T22:07:09.0997255Z ----------------------------------------------------------------------
2023-01-11T22:07:09.0997372Z Ran 1 test in 6.483s
2023-01-11T22:07:09.0997391Z 
2023-01-11T22:07:09.0997485Z OK
2023-01-11T22:07:09.0997504Z 
2023-01-11T22:07:09.0997628Z Generating XML reports...
2023-01-11T22:07:09.0998078Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220431.xml
2023-01-11T22:07:09.0998445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.0998624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.0999005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.0999178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.0999217Z 
2023-01-11T22:07:09.0999308Z Running tests...
2023-01-11T22:07:09.0999793Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1000200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1000470Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.1000489Z 
2023-01-11T22:07:09.1000744Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1000856Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1000875Z 
2023-01-11T22:07:09.1000982Z OK (skipped=1)
2023-01-11T22:07:09.1001001Z 
2023-01-11T22:07:09.1001125Z Generating XML reports...
2023-01-11T22:07:09.1001612Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220440.xml
2023-01-11T22:07:09.1001996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1002169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1002554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1002745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1002765Z 
2023-01-11T22:07:09.1002874Z Running tests...
2023-01-11T22:07:09.1003134Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1003450Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1003687Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.1003726Z 
2023-01-11T22:07:09.1003967Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1004080Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1004099Z 
2023-01-11T22:07:09.1004206Z OK (skipped=1)
2023-01-11T22:07:09.1004224Z 
2023-01-11T22:07:09.1004347Z Generating XML reports...
2023-01-11T22:07:09.1004801Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220443.xml
2023-01-11T22:07:09.1005168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1005342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1005720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1005897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1005932Z 
2023-01-11T22:07:09.1006024Z Running tests...
2023-01-11T22:07:09.1006282Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1006593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1006853Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.1006876Z 
2023-01-11T22:07:09.1007134Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1007246Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1007266Z 
2023-01-11T22:07:09.1007377Z OK (skipped=1)
2023-01-11T22:07:09.1007396Z 
2023-01-11T22:07:09.1007517Z Generating XML reports...
2023-01-11T22:07:09.1007944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220445.xml
2023-01-11T22:07:09.1008315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1008489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1008869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1009061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1009132Z 
2023-01-11T22:07:09.1009245Z Running tests...
2023-01-11T22:07:09.1009509Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1009821Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1010068Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.1010109Z 
2023-01-11T22:07:09.1010348Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1010462Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1010539Z 
2023-01-11T22:07:09.1010653Z OK (skipped=1)
2023-01-11T22:07:09.1010672Z 
2023-01-11T22:07:09.1010795Z Generating XML reports...
2023-01-11T22:07:09.1011248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220448.xml
2023-01-11T22:07:09.1011619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1011797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1012176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1012350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1012386Z 
2023-01-11T22:07:09.1012477Z Running tests...
2023-01-11T22:07:09.1012736Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1013052Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1013301Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1013521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38829
2023-01-11T22:07:09.1013739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38830
2023-01-11T22:07:09.1014111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1014287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1014648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1014841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1015209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1015382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1015755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1015940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1016186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1016433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1016817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1017216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1017449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1017676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1018010Z STAGE:2023-01-11 22:04:55 38829:38829 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1018335Z STAGE:2023-01-11 22:04:56 38830:38830 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1018727Z STAGE:2023-01-11 22:04:57 38829:38829 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1019076Z STAGE:2023-01-11 22:04:57 38829:38829 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1019408Z STAGE:2023-01-11 22:04:57 38830:38830 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1019736Z STAGE:2023-01-11 22:04:57 38830:38830 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1020109Z STAGE:2023-01-11 22:04:57 38829:38829 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1020443Z STAGE:2023-01-11 22:04:57 38830:38830 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1020775Z STAGE:2023-01-11 22:04:57 38830:38830 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1021117Z STAGE:2023-01-11 22:04:57 38830:38830 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1021449Z STAGE:2023-01-11 22:04:57 38829:38829 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1021790Z STAGE:2023-01-11 22:04:57 38829:38829 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1021896Z ok (7.873s)
2023-01-11T22:07:09.1021915Z 
2023-01-11T22:07:09.1022176Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1022272Z Ran 1 test in 7.873s
2023-01-11T22:07:09.1022291Z 
2023-01-11T22:07:09.1022381Z OK
2023-01-11T22:07:09.1022400Z 
2023-01-11T22:07:09.1022529Z Generating XML reports...
2023-01-11T22:07:09.1022982Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220450.xml
2023-01-11T22:07:09.1023353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1023529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1023917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1024111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1024130Z 
2023-01-11T22:07:09.1024241Z Running tests...
2023-01-11T22:07:09.1024486Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1024803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1025077Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1025302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38952
2023-01-11T22:07:09.1025522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38953
2023-01-11T22:07:09.1025892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1026073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1026451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1026624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1026987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1027162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1027535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1027725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1027971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1028269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1028670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1029069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1029282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1029553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1029897Z STAGE:2023-01-11 22:05:06 38953:38953 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1030219Z STAGE:2023-01-11 22:05:06 38952:38952 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1030552Z STAGE:2023-01-11 22:05:07 38952:38952 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1030904Z STAGE:2023-01-11 22:05:07 38952:38952 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1031238Z STAGE:2023-01-11 22:05:07 38953:38953 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1031582Z STAGE:2023-01-11 22:05:07 38953:38953 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1031909Z STAGE:2023-01-11 22:05:07 38953:38953 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1032220Z STAGE:2023-01-11 22:05:07 38952:38952 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1032549Z STAGE:2023-01-11 22:05:07 38953:38953 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1032890Z STAGE:2023-01-11 22:05:07 38953:38953 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1033219Z STAGE:2023-01-11 22:05:07 38952:38952 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1033564Z STAGE:2023-01-11 22:05:07 38952:38952 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1033669Z ok (7.868s)
2023-01-11T22:07:09.1033688Z 
2023-01-11T22:07:09.1033952Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1034066Z Ran 1 test in 7.868s
2023-01-11T22:07:09.1034085Z 
2023-01-11T22:07:09.1034159Z OK
2023-01-11T22:07:09.1034197Z 
2023-01-11T22:07:09.1034304Z Generating XML reports...
2023-01-11T22:07:09.1034756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220500.xml
2023-01-11T22:07:09.1035128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1035306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1035688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1035883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1035903Z 
2023-01-11T22:07:09.1036012Z Running tests...
2023-01-11T22:07:09.1036272Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1036568Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1036839Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.1036858Z 
2023-01-11T22:07:09.1037119Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1037232Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1037251Z 
2023-01-11T22:07:09.1037359Z OK (skipped=1)
2023-01-11T22:07:09.1037377Z 
2023-01-11T22:07:09.1037500Z Generating XML reports...
2023-01-11T22:07:09.1037948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220511.xml
2023-01-11T22:07:09.1038381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1038557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1038916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1039105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1039169Z 
2023-01-11T22:07:09.1039284Z Running tests...
2023-01-11T22:07:09.1039763Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1040084Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1040347Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:07:09.1040373Z 
2023-01-11T22:07:09.1040630Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1040740Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1040759Z 
2023-01-11T22:07:09.1040849Z OK (skipped=1)
2023-01-11T22:07:09.1040886Z 
2023-01-11T22:07:09.1040992Z Generating XML reports...
2023-01-11T22:07:09.1041441Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220513.xml
2023-01-11T22:07:09.1041818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1041992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1042373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1042563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1042583Z 
2023-01-11T22:07:09.1042696Z Running tests...
2023-01-11T22:07:09.1042959Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1043255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1043638Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:07:09.1043657Z 
2023-01-11T22:07:09.1043913Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1044024Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1044046Z 
2023-01-11T22:07:09.1044151Z OK (skipped=1)
2023-01-11T22:07:09.1044170Z 
2023-01-11T22:07:09.1044293Z Generating XML reports...
2023-01-11T22:07:09.1044738Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220516.xml
2023-01-11T22:07:09.1045108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1045287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1045651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1045842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1045861Z 
2023-01-11T22:07:09.1045969Z Running tests...
2023-01-11T22:07:09.1046229Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1046545Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1046809Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:07:09.1046829Z 
2023-01-11T22:07:09.1047088Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1047201Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1047220Z 
2023-01-11T22:07:09.1047409Z OK (skipped=1)
2023-01-11T22:07:09.1047429Z 
2023-01-11T22:07:09.1047536Z Generating XML reports...
2023-01-11T22:07:09.1047984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220518.xml
2023-01-11T22:07:09.1048349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1048524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1049001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1049201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1049221Z 
2023-01-11T22:07:09.1049330Z Running tests...
2023-01-11T22:07:09.1049593Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1049885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1050177Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T22:07:09.1050197Z 
2023-01-11T22:07:09.1050455Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1050566Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1050585Z 
2023-01-11T22:07:09.1050690Z OK (skipped=1)
2023-01-11T22:07:09.1050710Z 
2023-01-11T22:07:09.1050834Z Generating XML reports...
2023-01-11T22:07:09.1051282Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220520.xml
2023-01-11T22:07:09.1051651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1051825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1052181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1052375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1052395Z 
2023-01-11T22:07:09.1052505Z Running tests...
2023-01-11T22:07:09.1052763Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1053070Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1053384Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T22:07:09.1053404Z 
2023-01-11T22:07:09.1053662Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1053776Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1053795Z 
2023-01-11T22:07:09.1053903Z OK (skipped=1)
2023-01-11T22:07:09.1053923Z 
2023-01-11T22:07:09.1054027Z Generating XML reports...
2023-01-11T22:07:09.1054471Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220523.xml
2023-01-11T22:07:09.1054847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1055022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1055400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1055591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1055614Z 
2023-01-11T22:07:09.1055724Z Running tests...
2023-01-11T22:07:09.1055984Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1056295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1056584Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T22:07:09.1056660Z 
2023-01-11T22:07:09.1056924Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1057036Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1057055Z 
2023-01-11T22:07:09.1057162Z OK (skipped=1)
2023-01-11T22:07:09.1057181Z 
2023-01-11T22:07:09.1057304Z Generating XML reports...
2023-01-11T22:07:09.1057751Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220525.xml
2023-01-11T22:07:09.1058171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1058353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1058735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1058908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1058930Z 
2023-01-11T22:07:09.1059040Z Running tests...
2023-01-11T22:07:09.1059299Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1059612Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1059903Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:07:09.1059922Z 
2023-01-11T22:07:09.1060180Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1060293Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1060312Z 
2023-01-11T22:07:09.1060420Z OK (skipped=1)
2023-01-11T22:07:09.1060439Z 
2023-01-11T22:07:09.1060545Z Generating XML reports...
2023-01-11T22:07:09.1060995Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220528.xml
2023-01-11T22:07:09.1061359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1061537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1061916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1062104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1062123Z 
2023-01-11T22:07:09.1062232Z Running tests...
2023-01-11T22:07:09.1062493Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1062809Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1063044Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1063268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39339
2023-01-11T22:07:09.1063486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39340
2023-01-11T22:07:09.1063861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1064036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1064414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1064605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1064972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1065127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1065503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1065695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1066001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1066245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1066648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1067044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1067320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1067557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1067642Z ok (5.851s)
2023-01-11T22:07:09.1067661Z 
2023-01-11T22:07:09.1067928Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1068039Z Ran 1 test in 5.851s
2023-01-11T22:07:09.1068062Z 
2023-01-11T22:07:09.1068155Z OK
2023-01-11T22:07:09.1068174Z 
2023-01-11T22:07:09.1068300Z Generating XML reports...
2023-01-11T22:07:09.1068753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220530.xml
2023-01-11T22:07:09.1069126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1069303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1069689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1069863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1069882Z 
2023-01-11T22:07:09.1069993Z Running tests...
2023-01-11T22:07:09.1070254Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1070565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1070850Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1071067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39457
2023-01-11T22:07:09.1071283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39458
2023-01-11T22:07:09.1071651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1071810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1072189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1072379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1072741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1072919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1073293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1073477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1073722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1073972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1074355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1074754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1074985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1075272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1075611Z STAGE:2023-01-11 22:05:44 39458:39458 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1075932Z STAGE:2023-01-11 22:05:44 39457:39457 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1076268Z STAGE:2023-01-11 22:05:44 39458:39458 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1076647Z STAGE:2023-01-11 22:05:44 39457:39457 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1077002Z STAGE:2023-01-11 22:05:44 39457:39457 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1077330Z STAGE:2023-01-11 22:05:44 39458:39458 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1077433Z ok (6.459s)
2023-01-11T22:07:09.1077454Z 
2023-01-11T22:07:09.1077721Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1077833Z Ran 1 test in 6.459s
2023-01-11T22:07:09.1077852Z 
2023-01-11T22:07:09.1077944Z OK
2023-01-11T22:07:09.1077964Z 
2023-01-11T22:07:09.1078089Z Generating XML reports...
2023-01-11T22:07:09.1078537Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220538.xml
2023-01-11T22:07:09.1078904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1079065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1079656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1079857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1079878Z 
2023-01-11T22:07:09.1079987Z Running tests...
2023-01-11T22:07:09.1080261Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1080574Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1080851Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1081070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39579
2023-01-11T22:07:09.1081285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39580
2023-01-11T22:07:09.1081643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1081818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1082197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1082390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1082759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1082933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1083304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1083494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1083727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1083973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1084370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1084767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1085078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1085309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1085655Z STAGE:2023-01-11 22:05:53 39579:39579 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1085978Z STAGE:2023-01-11 22:05:53 39580:39580 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:07:09.1086364Z STAGE:2023-01-11 22:05:53 39580:39580 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1086694Z STAGE:2023-01-11 22:05:53 39579:39579 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:07:09.1087040Z STAGE:2023-01-11 22:05:53 39580:39580 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1087379Z STAGE:2023-01-11 22:05:53 39579:39579 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:07:09.1087487Z ok (6.539s)
2023-01-11T22:07:09.1087507Z 
2023-01-11T22:07:09.1087770Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1087883Z Ran 1 test in 6.539s
2023-01-11T22:07:09.1087902Z 
2023-01-11T22:07:09.1087995Z OK
2023-01-11T22:07:09.1088014Z 
2023-01-11T22:07:09.1088136Z Generating XML reports...
2023-01-11T22:07:09.1088587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220547.xml
2023-01-11T22:07:09.1088940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1089116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1089498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1089688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1089711Z 
2023-01-11T22:07:09.1089820Z Running tests...
2023-01-11T22:07:09.1090082Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1090393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1090680Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:07:09.1090700Z 
2023-01-11T22:07:09.1090963Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1091060Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1091079Z 
2023-01-11T22:07:09.1091189Z OK (skipped=1)
2023-01-11T22:07:09.1091208Z 
2023-01-11T22:07:09.1091332Z Generating XML reports...
2023-01-11T22:07:09.1091994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220556.xml
2023-01-11T22:07:09.1092386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1092561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1092939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1093132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1093150Z 
2023-01-11T22:07:09.1093242Z Running tests...
2023-01-11T22:07:09.1093508Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1093822Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1094213Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:07:09.1094233Z 
2023-01-11T22:07:09.1094492Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1094669Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1094688Z 
2023-01-11T22:07:09.1094795Z OK (skipped=1)
2023-01-11T22:07:09.1094821Z 
2023-01-11T22:07:09.1094939Z Generating XML reports...
2023-01-11T22:07:09.1095392Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220559.xml
2023-01-11T22:07:09.1095745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1095965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1096356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1096547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1096567Z 
2023-01-11T22:07:09.1096677Z Running tests...
2023-01-11T22:07:09.1096939Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1097256Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1097557Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:07:09.1097576Z 
2023-01-11T22:07:09.1097834Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1097927Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1097946Z 
2023-01-11T22:07:09.1098056Z OK (skipped=1)
2023-01-11T22:07:09.1098076Z 
2023-01-11T22:07:09.1098200Z Generating XML reports...
2023-01-11T22:07:09.1098647Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220601.xml
2023-01-11T22:07:09.1099014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1099190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1099572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1099763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1099783Z 
2023-01-11T22:07:09.1099892Z Running tests...
2023-01-11T22:07:09.1100135Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1100445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1100745Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:07:09.1100765Z 
2023-01-11T22:07:09.1101024Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1101138Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1101157Z 
2023-01-11T22:07:09.1101266Z OK (skipped=1)
2023-01-11T22:07:09.1101289Z 
2023-01-11T22:07:09.1101412Z Generating XML reports...
2023-01-11T22:07:09.1101858Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220604.xml
2023-01-11T22:07:09.1102209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1102387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1102768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1102959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1102979Z 
2023-01-11T22:07:09.1103085Z Running tests...
2023-01-11T22:07:09.1103345Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1103657Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1104033Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T22:07:09.1104053Z 
2023-01-11T22:07:09.1104313Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1104407Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1104445Z 
2023-01-11T22:07:09.1104535Z OK (skipped=1)
2023-01-11T22:07:09.1104554Z 
2023-01-11T22:07:09.1104677Z Generating XML reports...
2023-01-11T22:07:09.1105167Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220606.xml
2023-01-11T22:07:09.1105543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1105720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1106099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1106295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1106315Z 
2023-01-11T22:07:09.1106423Z Running tests...
2023-01-11T22:07:09.1106669Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1106981Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1107271Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T22:07:09.1107291Z 
2023-01-11T22:07:09.1107550Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1107663Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1107682Z 
2023-01-11T22:07:09.1107790Z OK (skipped=1)
2023-01-11T22:07:09.1107809Z 
2023-01-11T22:07:09.1107931Z Generating XML reports...
2023-01-11T22:07:09.1108373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220608.xml
2023-01-11T22:07:09.1108752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1108910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1109292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1109484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1109504Z 
2023-01-11T22:07:09.1109611Z Running tests...
2023-01-11T22:07:09.1109874Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1110191Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1110460Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1110682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39899
2023-01-11T22:07:09.1110886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39900
2023-01-11T22:07:09.1111254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1111425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1111802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1111993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1112354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1112526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1112896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1113142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1113372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1113616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1114021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1114459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1114701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1114931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1115034Z ok (7.372s)
2023-01-11T22:07:09.1115054Z 
2023-01-11T22:07:09.1115322Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1115419Z Ran 1 test in 7.373s
2023-01-11T22:07:09.1115457Z 
2023-01-11T22:07:09.1115531Z OK
2023-01-11T22:07:09.1115550Z 
2023-01-11T22:07:09.1115673Z Generating XML reports...
2023-01-11T22:07:09.1116122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220611.xml
2023-01-11T22:07:09.1116492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1116670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1117051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1117242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1117262Z 
2023-01-11T22:07:09.1117372Z Running tests...
2023-01-11T22:07:09.1117615Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1117934Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1118209Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T22:07:09.1118228Z 
2023-01-11T22:07:09.1118486Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1118600Z Ran 1 test in 0.002s
2023-01-11T22:07:09.1118619Z 
2023-01-11T22:07:09.1118728Z OK (skipped=1)
2023-01-11T22:07:09.1118747Z 
2023-01-11T22:07:09.1118873Z Generating XML reports...
2023-01-11T22:07:09.1119324Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220621.xml
2023-01-11T22:07:09.1119889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1120049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1120433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1120624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1120643Z 
2023-01-11T22:07:09.1120751Z Running tests...
2023-01-11T22:07:09.1121016Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1121329Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1121586Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1121807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40051
2023-01-11T22:07:09.1122007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40052
2023-01-11T22:07:09.1122377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1122637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1123020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1123210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1123572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1123799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1124183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1124374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1124602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1124852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1125253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1125650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1125879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1126113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1126217Z ok (6.846s)
2023-01-11T22:07:09.1126237Z 
2023-01-11T22:07:09.1126502Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1126597Z Ran 1 test in 6.847s
2023-01-11T22:07:09.1126635Z 
2023-01-11T22:07:09.1126710Z OK
2023-01-11T22:07:09.1126729Z 
2023-01-11T22:07:09.1126854Z Generating XML reports...
2023-01-11T22:07:09.1127310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220623.xml
2023-01-11T22:07:09.1127680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1127859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1128237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1128433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1128452Z 
2023-01-11T22:07:09.1128560Z Running tests...
2023-01-11T22:07:09.1128804Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1129116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1129408Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1129632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40166
2023-01-11T22:07:09.1129848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40167
2023-01-11T22:07:09.1130215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1130392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1130769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1130959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1131302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1131471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1131912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1132098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1132343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1132584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1133029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1133436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1133647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1133874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1134667Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:07:09.1135458Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:07:09.1135567Z ok (7.351s)
2023-01-11T22:07:09.1135587Z 
2023-01-11T22:07:09.1135859Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1135972Z Ran 1 test in 7.351s
2023-01-11T22:07:09.1135992Z 
2023-01-11T22:07:09.1136087Z OK
2023-01-11T22:07:09.1136105Z 
2023-01-11T22:07:09.1136230Z Generating XML reports...
2023-01-11T22:07:09.1136687Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220633.xml
2023-01-11T22:07:09.1137060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1137286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1137654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1137850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1137869Z 
2023-01-11T22:07:09.1137978Z Running tests...
2023-01-11T22:07:09.1138237Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1138549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1138833Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1139055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40285
2023-01-11T22:07:09.1139274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40286
2023-01-11T22:07:09.1139644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1139801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1140238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1140429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1140794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1140965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1141395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1141592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1141841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1142068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1142472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1142876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1143103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1143330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1143573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.1143816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.1144220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.1144611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.1144854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:09.1145071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:09.1145460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.1145852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.1145961Z ok (11.366s)
2023-01-11T22:07:09.1145981Z 
2023-01-11T22:07:09.1146246Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1146360Z Ran 1 test in 11.366s
2023-01-11T22:07:09.1146379Z 
2023-01-11T22:07:09.1146471Z OK
2023-01-11T22:07:09.1146490Z 
2023-01-11T22:07:09.1146615Z Generating XML reports...
2023-01-11T22:07:09.1147047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220643.xml
2023-01-11T22:07:09.1147425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1147602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1147983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1148174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1148197Z 
2023-01-11T22:07:09.1148305Z Running tests...
2023-01-11T22:07:09.1148567Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1148922Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:07:09.1149213Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:07:09.1149475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40407
2023-01-11T22:07:09.1149693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40408
2023-01-11T22:07:09.1150069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1150244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1150663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1150861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1151225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:07:09.1151400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:07:09.1151774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:07:09.1151948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:07:09.1152195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:07:09.1152439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:07:09.1152838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1153234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:07:09.1153465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:07:09.1153692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:07:09.1153933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:07:09.1154158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:07:09.1154556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.1154944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:07:09.1155184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:07:09.1155421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:07:09.1155816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.1156206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:07:09.1156314Z ok (11.238s)
2023-01-11T22:07:09.1156333Z 
2023-01-11T22:07:09.1156593Z ----------------------------------------------------------------------
2023-01-11T22:07:09.1156689Z Ran 1 test in 11.238s
2023-01-11T22:07:09.1156726Z 
2023-01-11T22:07:09.1156801Z OK
2023-01-11T22:07:09.1156821Z 
2023-01-11T22:07:09.1156945Z Generating XML reports...
2023-01-11T22:07:09.1157395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220656.xml
2023-01-11T22:07:09.1157417Z 
2023-01-11T22:07:09.1157861Z ##[endgroup]
2023-01-11T22:07:09.1158335Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_s9pi1hvh)
2023-01-11T22:07:09.1158355Z 
2023-01-11T22:07:09.1158483Z Shard 2: gloo should be run in 1
2023-01-11T22:07:09.1158606Z Shard 2: ucc should be run in 3
2023-01-11T22:07:09.1158934Z Running distributed/rpc/test_tensorpipe_agent ... [2023-01-11 22:07:08.916632]
2023-01-11T22:07:09.1159638Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:07:08.916863]
2023-01-11T22:07:11.5248895Z 
2023-01-11T22:07:11.5249708Z Expand the folded group to see the log file of distributed/rpc/test_tensorpipe_agent
2023-01-11T22:07:11.5251375Z ##[group]PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_5ncqjspb)
2023-01-11T22:07:11.5252041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4eyz548
2023-01-11T22:07:11.5252780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4eyz548/_remote_module_non_scriptable.py
2023-01-11T22:07:11.5253091Z 
2023-01-11T22:07:11.5253386Z ##[endgroup]
2023-01-11T22:07:11.5254134Z FINISHED PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_5ncqjspb)
2023-01-11T22:07:11.5254605Z 
2023-01-11T22:07:11.5255026Z Running distributed/pipeline/sync/test_transparency ... [2023-01-11 22:07:11.524933]
2023-01-11T22:07:11.5255669Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2023-01-11 22:07:11.525189]
2023-01-11T22:07:14.4667759Z 
2023-01-11T22:07:14.4668783Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency
2023-01-11T22:07:14.4670565Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_hl784j93)
2023-01-11T22:07:14.4671131Z ============================= test session starts ==============================
2023-01-11T22:07:14.4671744Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:07:14.4672114Z cachedir: .pytest_cache
2023-01-11T22:07:14.4672687Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:07:14.4673123Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:07:14.4673455Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:07:14.4674005Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:07:14.4674403Z collecting ... collected 1 item
2023-01-11T22:07:14.4674825Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears
2023-01-11T22:07:14.4675104Z 
2023-01-11T22:07:14.4675315Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2023-01-11T22:07:14.4675573Z 
2023-01-11T22:07:14.4675729Z ============================== 1 passed in 0.14s ===============================
2023-01-11T22:07:14.4675922Z 
2023-01-11T22:07:14.4676245Z ##[endgroup]
2023-01-11T22:07:14.4676891Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_hl784j93)
2023-01-11T22:07:14.4677290Z 
2023-01-11T22:07:14.4677566Z Running distributed/pipeline/sync/test_pipe ... [2023-01-11 22:07:14.466826]
2023-01-11T22:07:14.4678173Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2023-01-11 22:07:14.467085]
2023-01-11T22:07:22.1151332Z 
2023-01-11T22:07:22.1152119Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe
2023-01-11T22:07:22.1153082Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_ge8_d_zl)
2023-01-11T22:07:22.1153620Z ============================= test session starts ==============================
2023-01-11T22:07:22.1154243Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:07:22.1154898Z cachedir: .pytest_cache
2023-01-11T22:07:22.1155487Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:07:22.1155923Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:07:22.1156235Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:07:22.1156815Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:07:22.1157350Z collecting ... collected 56 items
2023-01-11T22:07:22.1164741Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper
2023-01-11T22:07:22.1170316Z 
2023-01-11T22:07:22.1170529Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2023-01-11T22:07:22.1170969Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2023-01-11T22:07:22.1171389Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2023-01-11T22:07:22.1171824Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2023-01-11T22:07:22.1172232Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2023-01-11T22:07:22.1172666Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2023-01-11T22:07:22.1173100Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2023-01-11T22:07:22.1173501Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2023-01-11T22:07:22.1173949Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2023-01-11T22:07:22.1174407Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2023-01-11T22:07:22.1174856Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2023-01-11T22:07:22.1175292Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2023-01-11T22:07:22.1175729Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2023-01-11T22:07:22.1176144Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2023-01-11T22:07:22.1176567Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2023-01-11T22:07:22.1177008Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2023-01-11T22:07:22.1177430Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2023-01-11T22:07:22.1177856Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2023-01-11T22:07:22.1178267Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2023-01-11T22:07:22.1178678Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2023-01-11T22:07:22.1179091Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2023-01-11T22:07:22.1179494Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2023-01-11T22:07:22.1179933Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2023-01-11T22:07:22.1180380Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2023-01-11T22:07:22.1180832Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2023-01-11T22:07:22.1181266Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2023-01-11T22:07:22.1181706Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2023-01-11T22:07:22.1182163Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2023-01-11T22:07:22.1182599Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2023-01-11T22:07:22.1183051Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2023-01-11T22:07:22.1183509Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2023-01-11T22:07:22.1184022Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2023-01-11T22:07:22.1184431Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2023-01-11T22:07:22.1184859Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2023-01-11T22:07:22.1185304Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2023-01-11T22:07:22.1185759Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2023-01-11T22:07:22.1186252Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2023-01-11T22:07:22.1186732Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2023-01-11T22:07:22.1187207Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2023-01-11T22:07:22.1187635Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2023-01-11T22:07:22.1188053Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2023-01-11T22:07:22.1188482Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions PASSED    [ 75%]
2023-01-11T22:07:22.1188905Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2023-01-11T22:07:22.1189302Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2023-01-11T22:07:22.1189714Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2023-01-11T22:07:22.1190159Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2023-01-11T22:07:22.1190611Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2023-01-11T22:07:22.1191083Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device PASSED [ 85%]
2023-01-11T22:07:22.1191541Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules PASSED [ 87%]
2023-01-11T22:07:22.1192031Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2023-01-11T22:07:22.1192491Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2023-01-11T22:07:22.1192932Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] PASSED [ 92%]
2023-01-11T22:07:22.1193377Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] PASSED [ 94%]
2023-01-11T22:07:22.1193826Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] PASSED [ 96%]
2023-01-11T22:07:22.1194277Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device PASSED  [ 98%]
2023-01-11T22:07:22.1194707Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper PASSED  [100%]
2023-01-11T22:07:22.1194955Z 
2023-01-11T22:07:22.1195113Z ============================== 56 passed in 5.17s ==============================
2023-01-11T22:07:22.1195303Z 
2023-01-11T22:07:22.1195617Z ##[endgroup]
2023-01-11T22:07:22.1196271Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_ge8_d_zl)
2023-01-11T22:07:22.1196642Z 
2023-01-11T22:07:22.1196921Z Running distributed/pipeline/sync/test_inplace ... [2023-01-11 22:07:22.115199]
2023-01-11T22:07:22.1197523Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2023-01-11 22:07:22.115474]
2023-01-11T22:07:24.6434757Z 
2023-01-11T22:07:24.6435667Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace
2023-01-11T22:07:24.6437553Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_yah620hq)
2023-01-11T22:07:24.6438615Z ============================= test session starts ==============================
2023-01-11T22:07:24.6439218Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:07:24.6440683Z cachedir: .pytest_cache
2023-01-11T22:07:24.6441782Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:07:24.6442204Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:07:24.6442538Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:07:24.6443119Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:07:24.6443640Z collecting ... collected 3 items
2023-01-11T22:07:24.6444280Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2023-01-11T22:07:24.6444771Z 
2023-01-11T22:07:24.6445002Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2023-01-11T22:07:24.6445479Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2023-01-11T22:07:24.6445944Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2023-01-11T22:07:24.6446199Z 
2023-01-11T22:07:24.6446349Z ========================= 1 passed, 2 xfailed in 0.18s =========================
2023-01-11T22:07:24.6446553Z 
2023-01-11T22:07:24.6448564Z ##[endgroup]
2023-01-11T22:07:24.6449269Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_yah620hq)
2023-01-11T22:07:24.6449703Z 
2023-01-11T22:07:24.6449987Z Running distributed/pipeline/sync/test_copy ... [2023-01-11 22:07:24.643485]
2023-01-11T22:07:24.6450584Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2023-01-11 22:07:24.643730]
2023-01-11T22:07:29.3523897Z 
2023-01-11T22:07:29.3524683Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy
2023-01-11T22:07:29.3526101Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_rfb3z560)
2023-01-11T22:07:29.3526744Z ============================= test session starts ==============================
2023-01-11T22:07:29.3527558Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:07:29.3527935Z cachedir: .pytest_cache
2023-01-11T22:07:29.3528607Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:07:29.3529138Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:07:29.3529473Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:07:29.3530020Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:07:29.3530436Z collecting ... collected 5 items
2023-01-11T22:07:29.3531204Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors
2023-01-11T22:07:29.3531822Z 
2023-01-11T22:07:29.3532040Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2023-01-11T22:07:29.3532481Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED   [ 40%]
2023-01-11T22:07:29.3532896Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED   [ 60%]
2023-01-11T22:07:29.3533326Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED  [ 80%]
2023-01-11T22:07:29.3533761Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2023-01-11T22:07:29.3534258Z 
2023-01-11T22:07:29.3534418Z ============================== 5 passed in 2.32s ===============================
2023-01-11T22:07:29.3534611Z 
2023-01-11T22:07:29.3534912Z ##[endgroup]
2023-01-11T22:07:29.3535552Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_rfb3z560)
2023-01-11T22:07:29.3535932Z 
2023-01-11T22:07:29.3536213Z Running distributed/pipeline/sync/test_balance ... [2023-01-11 22:07:29.352398]
2023-01-11T22:07:29.3536879Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2023-01-11 22:07:29.352706]
2023-01-11T22:07:38.7098755Z 
2023-01-11T22:07:38.7099557Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance
2023-01-11T22:07:38.7100959Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_izpjq5m2)
2023-01-11T22:07:38.7101517Z ============================= test session starts ==============================
2023-01-11T22:07:38.7102130Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:07:38.7102517Z cachedir: .pytest_cache
2023-01-11T22:07:38.7103066Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:07:38.7103495Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:07:38.7103845Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:07:38.7104408Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:07:38.7104797Z collecting ... collected 18 items
2023-01-11T22:07:38.7106919Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad
2023-01-11T22:07:38.7109004Z 
2023-01-11T22:07:38.7109211Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2023-01-11T22:07:38.7109672Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2023-01-11T22:07:38.7110168Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2023-01-11T22:07:38.7110674Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2023-01-11T22:07:38.7111125Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2023-01-11T22:07:38.7111578Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2023-01-11T22:07:38.7112062Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2023-01-11T22:07:38.7112765Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2023-01-11T22:07:38.7113222Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2023-01-11T22:07:38.7113680Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2023-01-11T22:07:38.7114139Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2023-01-11T22:07:38.7114669Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2023-01-11T22:07:38.7115125Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2023-01-11T22:07:38.7115605Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2023-01-11T22:07:38.7116042Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2023-01-11T22:07:38.7116485Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2023-01-11T22:07:38.7116940Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2023-01-11T22:07:38.7117382Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2023-01-11T22:07:38.7117630Z 
2023-01-11T22:07:38.7117785Z ======================== 16 passed, 2 skipped in 6.86s =========================
2023-01-11T22:07:38.7117989Z 
2023-01-11T22:07:38.7118326Z ##[endgroup]
2023-01-11T22:07:38.7118961Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_izpjq5m2)
2023-01-11T22:07:38.7119341Z 
2023-01-11T22:07:38.7119893Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2023-01-11 22:07:38.709876]
2023-01-11T22:07:38.7120540Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2023-01-11 22:07:38.710238]
2023-01-11T22:07:41.1453382Z 
2023-01-11T22:07:41.1454242Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T22:07:41.1455292Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_ulgggw2w)
2023-01-11T22:07:41.1455834Z ============================= test session starts ==============================
2023-01-11T22:07:41.1456441Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:07:41.1456811Z cachedir: .pytest_cache
2023-01-11T22:07:41.1457386Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:07:41.1457811Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:07:41.1458135Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:07:41.1458697Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:07:41.1459088Z collecting ... collected 7 items
2023-01-11T22:07:41.1460075Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none
2023-01-11T22:07:41.1460905Z 
2023-01-11T22:07:41.1461112Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2023-01-11T22:07:41.1461567Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2023-01-11T22:07:41.1462019Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2023-01-11T22:07:41.1462744Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2023-01-11T22:07:41.1463188Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2023-01-11T22:07:41.1463647Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2023-01-11T22:07:41.1464095Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2023-01-11T22:07:41.1464361Z 
2023-01-11T22:07:41.1464565Z ============================== 7 passed in 0.06s ===============================
2023-01-11T22:07:41.1464769Z 
2023-01-11T22:07:41.1465082Z ##[endgroup]
2023-01-11T22:07:41.1465760Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_ulgggw2w)
2023-01-11T22:07:41.1466164Z 
2023-01-11T22:07:41.1466484Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2023-01-11 22:07:41.145261]
2023-01-11T22:07:41.1467136Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2023-01-11 22:07:41.145507]
2023-01-11T22:07:43.5334351Z 
2023-01-11T22:07:43.5334922Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T22:07:43.5336141Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_cpiv1kmk)
2023-01-11T22:07:43.5336902Z ============================= test session starts ==============================
2023-01-11T22:07:43.5337965Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:07:43.5338637Z cachedir: .pytest_cache
2023-01-11T22:07:43.5339224Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:07:43.5339662Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:07:43.5339971Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:07:43.5340552Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:07:43.5340945Z collecting ... collected 6 items
2023-01-11T22:07:43.5341963Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace
2023-01-11T22:07:43.5342812Z 
2023-01-11T22:07:43.5343053Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2023-01-11T22:07:43.5343541Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2023-01-11T22:07:43.5344019Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2023-01-11T22:07:43.5344509Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2023-01-11T22:07:43.5345014Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2023-01-11T22:07:43.5345509Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2023-01-11T22:07:43.5345755Z 
2023-01-11T22:07:43.5345911Z ============================== 6 passed in 0.04s ===============================
2023-01-11T22:07:43.5346101Z 
2023-01-11T22:07:43.5346410Z ##[endgroup]
2023-01-11T22:07:43.5347130Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_cpiv1kmk)
2023-01-11T22:07:43.5347819Z 
2023-01-11T22:07:43.5348078Z Running distributed/elastic/timer/api_test ... [2023-01-11 22:07:43.533573]
2023-01-11T22:07:43.5348751Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:07:43.533821]
2023-01-11T22:07:45.3197954Z 
2023-01-11T22:07:45.3199108Z Expand the folded group to see the log file of distributed/elastic/timer/api_test
2023-01-11T22:07:45.3201303Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_h2a2n1q6)
2023-01-11T22:07:45.3202112Z 
2023-01-11T22:07:45.3202707Z ##[endgroup]
2023-01-11T22:07:45.3203624Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_h2a2n1q6)
2023-01-11T22:07:45.3204389Z 
2023-01-11T22:07:45.3204992Z Running distributed/_shard/test_sharder ... [2023-01-11 22:07:45.319817]
2023-01-11T22:07:45.3205742Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:07:45.320133]
2023-01-11T22:07:47.4705370Z 
2023-01-11T22:07:47.4705923Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2023-01-11T22:07:47.4707139Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_8hq54l1x)
2023-01-11T22:07:47.4707722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9d7_w2q
2023-01-11T22:07:47.4708284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9d7_w2q/_remote_module_non_scriptable.py
2023-01-11T22:07:47.4708576Z 
2023-01-11T22:07:47.4708867Z ##[endgroup]
2023-01-11T22:07:47.4709713Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_8hq54l1x)
2023-01-11T22:07:47.4710051Z 
2023-01-11T22:07:47.4710328Z Running distributed/test_nccl ... [2023-01-11 22:07:47.470593]
2023-01-11T22:07:47.4711212Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:07:47.470839]
2023-01-11T22:07:52.6320069Z 
2023-01-11T22:07:52.6320556Z Expand the folded group to see the log file of distributed/test_nccl
2023-01-11T22:07:52.6321451Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_l3za8fwu)
2023-01-11T22:07:52.6321779Z 
2023-01-11T22:07:52.6321888Z Running tests...
2023-01-11T22:07:52.6322387Z ----------------------------------------------------------------------
2023-01-11T22:07:52.6322932Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2023-01-11T22:07:52.6323369Z   test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (1.147s)
2023-01-11T22:07:52.6323768Z   test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2023-01-11T22:07:52.6324124Z   test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2023-01-11T22:07:52.6324504Z   test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2023-01-11T22:07:52.6324888Z   test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2023-01-11T22:07:52.6325264Z   test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2023-01-11T22:07:52.6325639Z   test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.002s)
2023-01-11T22:07:52.6326015Z   test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.002s)
2023-01-11T22:07:52.6326374Z   test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.002s)
2023-01-11T22:07:52.6326757Z   test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2023-01-11T22:07:52.6327425Z   test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2023-01-11T22:07:52.6327792Z   test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.001s)
2023-01-11T22:07:52.6327996Z 
2023-01-11T22:07:52.6328268Z ----------------------------------------------------------------------
2023-01-11T22:07:52.6328597Z Ran 12 tests in 1.180s
2023-01-11T22:07:52.6328755Z 
2023-01-11T22:07:52.6328852Z OK
2023-01-11T22:07:52.6328986Z 
2023-01-11T22:07:52.6329111Z Generating XML reports...
2023-01-11T22:07:52.6329758Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20230111220750.xml
2023-01-11T22:07:52.6330093Z 
2023-01-11T22:07:52.6330403Z ##[endgroup]
2023-01-11T22:07:52.6330949Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_l3za8fwu)
2023-01-11T22:07:52.6331245Z 
2023-01-11T22:07:52.6331521Z Running distributed/checkpoint/test_traverse ... [2023-01-11 22:07:52.632070]
2023-01-11T22:07:52.6332219Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:07:52.632336]
2023-01-11T22:07:56.5894966Z 
2023-01-11T22:07:56.5895511Z Expand the folded group to see the log file of distributed/checkpoint/test_traverse
2023-01-11T22:07:56.5896903Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_z41gsza9)
2023-01-11T22:07:56.5897554Z 
2023-01-11T22:07:56.5897742Z Running tests...
2023-01-11T22:07:56.5898270Z ----------------------------------------------------------------------
2023-01-11T22:07:56.5898867Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_traverse
2023-01-11T22:07:56.5899365Z   test_get_element (__main__.TestTraverse) ... ok (1.650s)
2023-01-11T22:07:56.5899980Z   test_set_element (__main__.TestTraverse) ... ok (0.002s)
2023-01-11T22:07:56.5900407Z   test_traverse_doesnt_ignore_intermediate_collections (__main__.TestTraverse) ... ok (0.002s)
2023-01-11T22:07:56.5900806Z   test_traverse_nested_dict (__main__.TestTraverse) ... ok (0.001s)
2023-01-11T22:07:56.5901196Z   test_traverse_nested_list (__main__.TestTraverse) ... ok (0.002s)
2023-01-11T22:07:56.5901568Z   test_traverse_shallow (__main__.TestTraverse) ... ok (0.002s)
2023-01-11T22:07:56.5901921Z   test_traverse_with_ordered_dict (__main__.TestTraverse) ... ok (0.001s)
2023-01-11T22:07:56.5902152Z 
2023-01-11T22:07:56.5902436Z ----------------------------------------------------------------------
2023-01-11T22:07:56.5902769Z Ran 7 tests in 1.661s
2023-01-11T22:07:56.5902930Z 
2023-01-11T22:07:56.5903025Z OK
2023-01-11T22:07:56.5903141Z 
2023-01-11T22:07:56.5903271Z Generating XML reports...
2023-01-11T22:07:56.5903880Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20230111220754.xml
2023-01-11T22:07:56.5904223Z 
2023-01-11T22:07:56.5904551Z ##[endgroup]
2023-01-11T22:07:56.5905159Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_z41gsza9)
2023-01-11T22:07:56.5905526Z 
2023-01-11T22:07:56.5905804Z Running distributed/_composable/test_contract ... [2023-01-11 22:07:56.589529]
2023-01-11T22:07:56.5906507Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:07:56.589792]
2023-01-11T22:08:00.5469540Z 
2023-01-11T22:08:00.5470121Z Expand the folded group to see the log file of distributed/_composable/test_contract
2023-01-11T22:08:00.5471227Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_glvagkfr)
2023-01-11T22:08:00.5471588Z 
2023-01-11T22:08:00.5471703Z Running tests...
2023-01-11T22:08:00.5472203Z ----------------------------------------------------------------------
2023-01-11T22:08:00.5473292Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract
2023-01-11T22:08:00.5473729Z   test_add_hooks (__main__.TestContract) ... ok (1.638s)
2023-01-11T22:08:00.5474059Z   test_modify_fqn (__main__.TestContract) ... ok (0.002s)
2023-01-11T22:08:00.5474400Z   test_registry (__main__.TestContract) ... ok (0.002s)
2023-01-11T22:08:00.5474732Z   test_state (__main__.TestContract) ... ok (0.002s)
2023-01-11T22:08:00.5474991Z 
2023-01-11T22:08:00.5475630Z ----------------------------------------------------------------------
2023-01-11T22:08:00.5476374Z Ran 4 tests in 1.645s
2023-01-11T22:08:00.5476638Z 
2023-01-11T22:08:00.5476806Z OK
2023-01-11T22:08:00.5477016Z 
2023-01-11T22:08:00.5477251Z Generating XML reports...
2023-01-11T22:08:00.5478287Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20230111220758.xml
2023-01-11T22:08:00.5478923Z 
2023-01-11T22:08:00.5479739Z ##[endgroup]
2023-01-11T22:08:00.5480967Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_glvagkfr)
2023-01-11T22:08:00.5481750Z 
2023-01-11T22:08:00.5482354Z Running distributed/checkpoint/test_dedup_tensors ... [2023-01-11 22:08:00.547064]
2023-01-11T22:08:00.5483769Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:00.547334]
2023-01-11T22:08:04.4701628Z 
2023-01-11T22:08:04.4702360Z Expand the folded group to see the log file of distributed/checkpoint/test_dedup_tensors
2023-01-11T22:08:04.4703688Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_k_69leks)
2023-01-11T22:08:04.4704186Z 
2023-01-11T22:08:04.4704383Z Running tests...
2023-01-11T22:08:04.4704904Z ----------------------------------------------------------------------
2023-01-11T22:08:04.4705732Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors
2023-01-11T22:08:04.4706791Z   test_dedup_shards (__main__.TestDedupTensor) ... 2023-01-11 22:08:04,067 _dedup_tensors.py:44 INFO p:MainProcess t:MainThread: Duplicate keys to remove: {1: [MetadataIndex(fqn='tensor_0', offset=torch.Size([0]), index=None)]}
2023-01-11T22:08:04.4707328Z ok (1.613s)
2023-01-11T22:08:04.4707460Z 
2023-01-11T22:08:04.4707746Z ----------------------------------------------------------------------
2023-01-11T22:08:04.4708337Z Ran 1 test in 1.614s
2023-01-11T22:08:04.4708508Z 
2023-01-11T22:08:04.4708594Z OK
2023-01-11T22:08:04.4708729Z 
2023-01-11T22:08:04.4708837Z Generating XML reports...
2023-01-11T22:08:04.4709463Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20230111220802.xml
2023-01-11T22:08:04.4709827Z 
2023-01-11T22:08:04.4710143Z ##[endgroup]
2023-01-11T22:08:04.4710764Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_k_69leks)
2023-01-11T22:08:04.4711147Z 
2023-01-11T22:08:04.4711432Z Running distributed/checkpoint/test_planner ... [2023-01-11 22:08:04.470207]
2023-01-11T22:08:04.4712126Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:04.470470]
2023-01-11T22:08:08.3712361Z 
2023-01-11T22:08:08.3713074Z Expand the folded group to see the log file of distributed/checkpoint/test_planner
2023-01-11T22:08:08.3714446Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_sobejws6)
2023-01-11T22:08:08.3714868Z 
2023-01-11T22:08:08.3714992Z Running tests...
2023-01-11T22:08:08.3715779Z ----------------------------------------------------------------------
2023-01-11T22:08:08.3716749Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner
2023-01-11T22:08:08.3717214Z   test_global_plan (__main__.TestSavePlan) ... ok (1.628s)
2023-01-11T22:08:08.3717588Z   test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s)
2023-01-11T22:08:08.3717991Z   test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T22:08:08.3718382Z   test_local_load_plan (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T22:08:08.3718838Z   test_local_plan (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T22:08:08.3719057Z 
2023-01-11T22:08:08.3719340Z ----------------------------------------------------------------------
2023-01-11T22:08:08.3720341Z Ran 5 tests in 1.642s
2023-01-11T22:08:08.3720633Z 
2023-01-11T22:08:08.3720768Z OK
2023-01-11T22:08:08.3721000Z 
2023-01-11T22:08:08.3721209Z Generating XML reports...
2023-01-11T22:08:08.3721893Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20230111220806.xml
2023-01-11T22:08:08.3722243Z 
2023-01-11T22:08:08.3722562Z ##[endgroup]
2023-01-11T22:08:08.3723186Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_sobejws6)
2023-01-11T22:08:08.3723571Z 
2023-01-11T22:08:08.3723858Z Running distributed/fsdp/test_checkpoint_wrapper ... [2023-01-11 22:08:08.371244]
2023-01-11T22:08:08.3724569Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:08.371507]
2023-01-11T22:08:13.3675455Z 
2023-01-11T22:08:13.3675967Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper
2023-01-11T22:08:13.3676951Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_0nu93evc)
2023-01-11T22:08:13.3677634Z 
2023-01-11T22:08:13.3677762Z Running tests...
2023-01-11T22:08:13.3678285Z ----------------------------------------------------------------------
2023-01-11T22:08:13.3678869Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2023-01-11T22:08:13.3679734Z   test_apply_activation_checkpointing (__main__.CheckpointWrapperTest)
2023-01-11T22:08:13.3680387Z Ensures that `apply_activation_checkpointing` can be used ... ok (1.766s)
2023-01-11T22:08:13.3680846Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... ok (0.417s)
2023-01-11T22:08:13.3681315Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.009s)
2023-01-11T22:08:13.3681739Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2023-01-11T22:08:13.3682147Z Tests that using checkpoint_wrapper or the functional ... ok (0.489s)
2023-01-11T22:08:13.3682578Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s)
2023-01-11T22:08:13.3682969Z   test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s)
2023-01-11T22:08:13.3683388Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.003s)
2023-01-11T22:08:13.3683655Z 
2023-01-11T22:08:13.3683937Z ----------------------------------------------------------------------
2023-01-11T22:08:13.3684262Z Ran 7 tests in 2.687s
2023-01-11T22:08:13.3684424Z 
2023-01-11T22:08:13.3684500Z OK
2023-01-11T22:08:13.3684633Z 
2023-01-11T22:08:13.3684763Z Generating XML reports...
2023-01-11T22:08:13.3685410Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20230111220810.xml
2023-01-11T22:08:13.3685789Z 
2023-01-11T22:08:13.3686089Z ##[endgroup]
2023-01-11T22:08:13.3686721Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_0nu93evc)
2023-01-11T22:08:13.3687350Z 
2023-01-11T22:08:13.3687681Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2023-01-11 22:08:13.367621]
2023-01-11T22:08:13.3688444Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:13.367886]
2023-01-11T22:08:19.7076001Z 
2023-01-11T22:08:19.7076565Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T22:08:19.7077999Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_qgxd7om2)
2023-01-11T22:08:19.7078436Z 
2023-01-11T22:08:19.7078539Z Running tests...
2023-01-11T22:08:19.7079063Z ----------------------------------------------------------------------
2023-01-11T22:08:19.7080007Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2023-01-11T22:08:19.7080572Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:08:19.7081091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43758
2023-01-11T22:08:19.7081538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43759
2023-01-11T22:08:19.7081979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 43760
2023-01-11T22:08:19.7082405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 43761
2023-01-11T22:08:19.7083049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:19.7083477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:19.7084054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:19.7084512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:19.7085090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:19.7085528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:19.7086080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:19.7086544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:19.7087119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:19.7087555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:19.7088105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:19.7088575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:19.7089146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:19.7089566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:19.7090155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:19.7090634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:19.7091051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:08:19.7091502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:08:19.7091970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:08:19.7092424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:08:19.7092942Z skip: Need at least 4 CUDA devices (4.034s)
2023-01-11T22:08:19.7093120Z 
2023-01-11T22:08:19.7093397Z ----------------------------------------------------------------------
2023-01-11T22:08:19.7093719Z Ran 1 test in 4.034s
2023-01-11T22:08:19.7093880Z 
2023-01-11T22:08:19.7093988Z OK (skipped=1)
2023-01-11T22:08:19.7094148Z 
2023-01-11T22:08:19.7094267Z Generating XML reports...
2023-01-11T22:08:19.7095013Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20230111220815.xml
2023-01-11T22:08:19.7095451Z 
2023-01-11T22:08:19.7095771Z ##[endgroup]
2023-01-11T22:08:19.7096450Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_qgxd7om2)
2023-01-11T22:08:19.7096861Z 
2023-01-11T22:08:19.7097185Z Running distributed/tensor/parallel/test_view_sharding_dim_change ... [2023-01-11 22:08:19.707660]
2023-01-11T22:08:19.7097937Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_view_sharding_dim_change.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:19.707917]
2023-01-11T22:08:27.5197771Z 
2023-01-11T22:08:27.5198360Z Expand the folded group to see the log file of distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T22:08:27.5199901Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_view_sharding_dim_change_xg0fz1dg)
2023-01-11T22:08:27.5200358Z 
2023-01-11T22:08:27.5200459Z Running tests...
2023-01-11T22:08:27.5200962Z ----------------------------------------------------------------------
2023-01-11T22:08:27.5201580Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change
2023-01-11T22:08:27.5202188Z   test_view_with_sharding_dim_change (__main__.TPViewShardingDimChangeTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:08:27.5202678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43929
2023-01-11T22:08:27.5203125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43930
2023-01-11T22:08:27.5203747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:27.5204187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:27.5204783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:27.5205252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:27.5205831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:27.5206260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:27.5206832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:27.5207290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:27.5207726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:08:27.5208199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:08:27.5208685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:08:27.5209167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:08:27.5209802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:27.5210492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:27.5211163Z ok (5.534s)
2023-01-11T22:08:27.5211310Z 
2023-01-11T22:08:27.5211585Z ----------------------------------------------------------------------
2023-01-11T22:08:27.5211892Z Ran 1 test in 5.534s
2023-01-11T22:08:27.5212051Z 
2023-01-11T22:08:27.5212144Z OK
2023-01-11T22:08:27.5212274Z 
2023-01-11T22:08:27.5212398Z Generating XML reports...
2023-01-11T22:08:27.5213178Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20230111220821.xml
2023-01-11T22:08:27.5213618Z 
2023-01-11T22:08:27.5213941Z ##[endgroup]
2023-01-11T22:08:27.5214656Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_view_sharding_dim_change_xg0fz1dg)
2023-01-11T22:08:27.5215079Z 
2023-01-11T22:08:27.5215382Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2023-01-11 22:08:27.519792]
2023-01-11T22:08:27.5216078Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:27.520118]
2023-01-11T22:08:35.8819640Z 
2023-01-11T22:08:35.8820265Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T22:08:35.8821367Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_gtxi6wun)
2023-01-11T22:08:35.8821779Z 
2023-01-11T22:08:35.8821968Z Running tests...
2023-01-11T22:08:35.8822796Z ----------------------------------------------------------------------
2023-01-11T22:08:35.8823405Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2023-01-11T22:08:35.8823870Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2023-01-11T22:08:35.8824348Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:08:35.8825136Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44043
2023-01-11T22:08:35.8825592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44044
2023-01-11T22:08:35.8826220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:35.8826657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:35.8827233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:35.8827699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:35.8828273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:35.8828700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:35.8829267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:35.8829723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:35.8830169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:08:35.8830647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:08:35.8831321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:35.8832007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:35.8832520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:08:35.8833242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:08:35.8834611Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:08:35.8835405Z   warnings.warn(
2023-01-11T22:08:35.8836565Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:08:35.8837326Z   warnings.warn(
2023-01-11T22:08:35.8837557Z dist init r=1, world=2
2023-01-11T22:08:35.8837803Z dist init r=0, world=2
2023-01-11T22:08:35.8838039Z ok (6.041s)
2023-01-11T22:08:35.8838184Z 
2023-01-11T22:08:35.8838453Z ----------------------------------------------------------------------
2023-01-11T22:08:35.8838757Z Ran 1 test in 6.041s
2023-01-11T22:08:35.8838916Z 
2023-01-11T22:08:35.8839007Z OK
2023-01-11T22:08:35.8839140Z 
2023-01-11T22:08:35.8839265Z Generating XML reports...
2023-01-11T22:08:35.8840178Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20230111220829.xml
2023-01-11T22:08:35.8840550Z 
2023-01-11T22:08:35.8840871Z ##[endgroup]
2023-01-11T22:08:35.8841514Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_gtxi6wun)
2023-01-11T22:08:35.8841902Z 
2023-01-11T22:08:35.8842180Z Running distributed/checkpoint/test_fsdp_model_state ... [2023-01-11 22:08:35.882050]
2023-01-11T22:08:35.8842890Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:35.882323]
2023-01-11T22:08:44.5940679Z 
2023-01-11T22:08:44.5941260Z Expand the folded group to see the log file of distributed/checkpoint/test_fsdp_model_state
2023-01-11T22:08:44.5942374Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_fsdp_model_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_model_state_1wqgdmzq)
2023-01-11T22:08:44.5942767Z 
2023-01-11T22:08:44.5942887Z Running tests...
2023-01-11T22:08:44.5943557Z ----------------------------------------------------------------------
2023-01-11T22:08:44.5944229Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_fsdp_model_state
2023-01-11T22:08:44.5945063Z   test_fsdp_model_state_no_resharding (__main__.FsdpModelStateCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:08:44.5945913Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44161
2023-01-11T22:08:44.5946418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44162
2023-01-11T22:08:44.5947251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:44.5947784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:44.5948369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:44.5948844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:44.5949433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:44.5950141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:44.5950718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:44.5951184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:44.5951601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:08:44.5952195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:08:44.5952696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:08:44.5953182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:08:44.5953828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:44.5954517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:44.5954957Z skip: Need at least 4 CUDA devices (3.915s)
2023-01-11T22:08:44.5955471Z   test_fsdp_model_state_with_resharding (__main__.FsdpModelStateCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44231
2023-01-11T22:08:44.5956040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44232
2023-01-11T22:08:44.5956711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:44.5957163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:44.5957734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:44.5958184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:44.5958766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:44.5959209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:44.5960061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:44.5960525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:44.5960968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:08:44.5961456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:08:44.5961920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:08:44.5962401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:08:44.5963062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:44.5963751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:08:44.5964173Z skip: Need at least 4 CUDA devices (2.407s)
2023-01-11T22:08:44.5964366Z 
2023-01-11T22:08:44.5964633Z ----------------------------------------------------------------------
2023-01-11T22:08:44.5964959Z Ran 2 tests in 6.322s
2023-01-11T22:08:44.5965120Z 
2023-01-11T22:08:44.5965212Z OK (skipped=2)
2023-01-11T22:08:44.5965370Z 
2023-01-11T22:08:44.5965494Z Generating XML reports...
2023-01-11T22:08:44.5966149Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_fsdp_model_state/TEST-FsdpModelStateCheckpoint-20230111220837.xml
2023-01-11T22:08:44.5966540Z 
2023-01-11T22:08:44.5966852Z ##[endgroup]
2023-01-11T22:08:44.5967499Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_fsdp_model_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_model_state_1wqgdmzq)
2023-01-11T22:08:44.5968013Z 
2023-01-11T22:08:44.5968318Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2023-01-11 22:08:44.594121]
2023-01-11T22:08:44.5969034Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:44.594377]
2023-01-11T22:08:53.2386865Z 
2023-01-11T22:08:53.2387708Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_chunk
2023-01-11T22:08:53.2388821Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_sz8x8_zv)
2023-01-11T22:08:53.2389199Z 
2023-01-11T22:08:53.2389311Z Running tests...
2023-01-11T22:08:53.2389823Z ----------------------------------------------------------------------
2023-01-11T22:08:53.2390982Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2023-01-11T22:08:53.2391556Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:08:53.2392034Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44336
2023-01-11T22:08:53.2392498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44337
2023-01-11T22:08:53.2392939Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44338
2023-01-11T22:08:53.2393390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44339
2023-01-11T22:08:53.2394035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2394442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2395018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2395488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2396064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2396509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2397055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2397517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2398076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2398513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2399076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2399931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2400501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2400943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2401511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2401958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2402392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:08:53.2402861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:08:53.2403320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:08:53.2403793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:08:53.2404299Z skip: Need at least 4 CUDA devices (3.948s)
2023-01-11T22:08:53.2404788Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44472
2023-01-11T22:08:53.2405329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44473
2023-01-11T22:08:53.2405752Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44474
2023-01-11T22:08:53.2406252Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44475
2023-01-11T22:08:53.2406877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2407302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2407874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2408344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2408916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2409340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2409921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2410360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2410918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2411352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2411913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2412376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2412927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:08:53.2413367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:08:53.2413927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:08:53.2414385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:08:53.2414824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:08:53.2415271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:08:53.2415734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:08:53.2416175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:08:53.2416558Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:08:53.2416753Z 
2023-01-11T22:08:53.2417026Z ----------------------------------------------------------------------
2023-01-11T22:08:53.2417380Z Ran 2 tests in 6.358s
2023-01-11T22:08:53.2417501Z 
2023-01-11T22:08:53.2417610Z OK (skipped=2)
2023-01-11T22:08:53.2417760Z 
2023-01-11T22:08:53.2417884Z Generating XML reports...
2023-01-11T22:08:53.2418561Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20230111220846.xml
2023-01-11T22:08:53.2418932Z 
2023-01-11T22:08:53.2419243Z ##[endgroup]
2023-01-11T22:08:53.2419897Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_sz8x8_zv)
2023-01-11T22:08:53.2420276Z 
2023-01-11T22:08:53.2420601Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2023-01-11 22:08:53.238760]
2023-01-11T22:08:53.2421450Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:08:53.239038]
2023-01-11T22:09:01.8844170Z 
2023-01-11T22:09:01.8845036Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T22:09:01.8846699Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_e7v4kn3_)
2023-01-11T22:09:01.8847153Z 
2023-01-11T22:09:01.8847318Z Running tests...
2023-01-11T22:09:01.8848352Z ----------------------------------------------------------------------
2023-01-11T22:09:01.8849246Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2023-01-11T22:09:01.8849790Z   test_sharded_tensor_reshard (__main__.TestReshard) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:09:01.8850244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44643
2023-01-11T22:09:01.8850693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44644
2023-01-11T22:09:01.8851141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44645
2023-01-11T22:09:01.8851749Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44646
2023-01-11T22:09:01.8853160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8854081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8854670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8855123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8855743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8856190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8856756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8857207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8857782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8858278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8858848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8859293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8859862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8860309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8860856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8861314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8861746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:01.8862228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:01.8862669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:01.8863127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:01.8863515Z skip: Need at least 4 CUDA devices (4.039s)
2023-01-11T22:09:01.8863971Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44779
2023-01-11T22:09:01.8864650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44780
2023-01-11T22:09:01.8865093Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44781
2023-01-11T22:09:01.8865528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44782
2023-01-11T22:09:01.8866225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8866682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8867262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8867707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8868279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8868728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8869295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8869741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8870314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8870758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8871370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8871837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8872438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:01.8872880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:01.8873456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:01.8873936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:01.8874370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:01.8874862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:01.8875325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:01.8875799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:01.8876209Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:09:01.8876401Z 
2023-01-11T22:09:01.8876679Z ----------------------------------------------------------------------
2023-01-11T22:09:01.8877022Z Ran 2 tests in 6.349s
2023-01-11T22:09:01.8877195Z 
2023-01-11T22:09:01.8877304Z OK (skipped=2)
2023-01-11T22:09:01.8877456Z 
2023-01-11T22:09:01.8877580Z Generating XML reports...
2023-01-11T22:09:01.8878212Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20230111220855.xml
2023-01-11T22:09:01.8878572Z 
2023-01-11T22:09:01.8878905Z ##[endgroup]
2023-01-11T22:09:01.8879968Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_e7v4kn3_)
2023-01-11T22:09:01.8880417Z 
2023-01-11T22:09:01.8880687Z Running distributed/_tensor/test_view_ops ... [2023-01-11 22:09:01.884504]
2023-01-11T22:09:01.8881370Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:09:01.884770]
2023-01-11T22:09:10.7483160Z 
2023-01-11T22:09:10.7483810Z Expand the folded group to see the log file of distributed/_tensor/test_view_ops
2023-01-11T22:09:10.7485285Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_hahwv_0j)
2023-01-11T22:09:10.7486111Z 
2023-01-11T22:09:10.7486323Z Running tests...
2023-01-11T22:09:10.7487180Z ----------------------------------------------------------------------
2023-01-11T22:09:10.7488629Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_view_ops
2023-01-11T22:09:10.7489191Z   test_view_groups (__main__.TestViewOps) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:09:10.7490050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44950
2023-01-11T22:09:10.7490970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44951
2023-01-11T22:09:10.7491908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 44952
2023-01-11T22:09:10.7492597Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 44953
2023-01-11T22:09:10.7493029Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 44954
2023-01-11T22:09:10.7493466Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 44955
2023-01-11T22:09:10.7494088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7494544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7495121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7495575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7496150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7496592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7497160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7497604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7498175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7498611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7499183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7499629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7500199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7500639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7501188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7501644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7502215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7502651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7503205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7503664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7504231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7504648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7505370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7505826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7506256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:10.7506713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:10.7507263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:10.7507736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:10.7508197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:09:10.7508640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:09:10.7508975Z ok (4.126s)
2023-01-11T22:09:10.7509381Z   test_view_ops (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45154
2023-01-11T22:09:10.7509874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45155
2023-01-11T22:09:10.7510302Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 45156
2023-01-11T22:09:10.7510736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 45157
2023-01-11T22:09:10.7511171Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 45158
2023-01-11T22:09:10.7511594Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 45159
2023-01-11T22:09:10.7512201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7512649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7513217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7513669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7514240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7514680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7515233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7515694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7516265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7516701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7517250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7517714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7518281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7518718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7519263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7520007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7520588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7521008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7521574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7522132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7522703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:10.7523121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:10.7523683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:10.7524135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:10.7524616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:10.7525104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:09:10.7525557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:10.7526026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:10.7526473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:10.7526924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:09:10.7527310Z skip: Need at least 6 CUDA devices (2.419s)
2023-01-11T22:09:10.7527505Z 
2023-01-11T22:09:10.7527764Z ----------------------------------------------------------------------
2023-01-11T22:09:10.7528089Z Ran 2 tests in 6.545s
2023-01-11T22:09:10.7528252Z 
2023-01-11T22:09:10.7528360Z OK (skipped=1)
2023-01-11T22:09:10.7528519Z 
2023-01-11T22:09:10.7528644Z Generating XML reports...
2023-01-11T22:09:10.7529196Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20230111220903.xml
2023-01-11T22:09:10.7529521Z 
2023-01-11T22:09:10.7529851Z ##[endgroup]
2023-01-11T22:09:10.7530434Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_hahwv_0j)
2023-01-11T22:09:10.7530783Z 
2023-01-11T22:09:10.7531059Z Running distributed/fsdp/test_distributed_checkpoint ... [2023-01-11 22:09:10.748389]
2023-01-11T22:09:10.7531774Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:09:10.748649]
2023-01-11T22:09:22.8857603Z 
2023-01-11T22:09:22.8858467Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2023-01-11T22:09:22.8860229Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_wzuxl3b1)
2023-01-11T22:09:22.8860975Z 
2023-01-11T22:09:22.8861218Z Running tests...
2023-01-11T22:09:22.8862243Z ----------------------------------------------------------------------
2023-01-11T22:09:22.8862824Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2023-01-11T22:09:22.8863549Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:09:22.8864388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45393
2023-01-11T22:09:22.8864841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45394
2023-01-11T22:09:22.8865736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:22.8866608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:22.8867847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:22.8868326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:22.8868892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:22.8869611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:22.8870184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:22.8870629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:22.8871081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:09:22.8871667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:09:22.8872340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:09:22.8873008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:09:22.8873526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:22.8874000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:22.8874352Z dist init r=0, world=2
2023-01-11T22:09:22.8874583Z dist init r=1, world=2
2023-01-11T22:09:22.8874820Z ok (5.661s)
2023-01-11T22:09:22.8875338Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45476
2023-01-11T22:09:22.8875927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45477
2023-01-11T22:09:22.8876533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:22.8876977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:22.8877552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:22.8878004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:22.8878582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:22.8879019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:22.8880072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:22.8880558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:22.8881010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:09:22.8881502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:09:22.8882150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:09:22.8882837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:09:22.8883352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:22.8883818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:22.8884153Z dist init r=1, world=2
2023-01-11T22:09:22.8884401Z dist init r=0, world=2
2023-01-11T22:09:22.8884637Z ok (4.111s)
2023-01-11T22:09:22.8884767Z 
2023-01-11T22:09:22.8885064Z ----------------------------------------------------------------------
2023-01-11T22:09:22.8885393Z Ran 2 tests in 9.772s
2023-01-11T22:09:22.8885552Z 
2023-01-11T22:09:22.8885645Z OK
2023-01-11T22:09:22.8885775Z 
2023-01-11T22:09:22.8885882Z Generating XML reports...
2023-01-11T22:09:22.8886536Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20230111220912.xml
2023-01-11T22:09:22.8887051Z 
2023-01-11T22:09:22.8887385Z ##[endgroup]
2023-01-11T22:09:22.8888020Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_wzuxl3b1)
2023-01-11T22:09:22.8888410Z 
2023-01-11T22:09:22.8888702Z Running distributed/elastic/timer/local_timer_example ... [2023-01-11 22:09:22.885797]
2023-01-11T22:09:22.8889493Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:09:22.886224]
2023-01-11T22:09:36.1481091Z 
2023-01-11T22:09:36.1482024Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example
2023-01-11T22:09:36.1483790Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_z8xif160)
2023-01-11T22:09:36.1484648Z 
2023-01-11T22:09:36.1484936Z Running tests...
2023-01-11T22:09:36.1485858Z ----------------------------------------------------------------------
2023-01-11T22:09:36.1486952Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2023-01-11T22:09:36.1487940Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2023-01-11 22:09:26,345 driver: init
2023-01-11T22:09:36.1489206Z [INFO] 2023-01-11 22:09:26,363 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2023-01-11T22:09:36.1490041Z [INFO] 2023-01-11 22:09:26,363 api: Starting watchdog thread...
2023-01-11T22:09:36.1490995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1491946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1492970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1494022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1494660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1495100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1495665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1496127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1496684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1497125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1497689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1498151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1498704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1499143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1499703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1500142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1500715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1501152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1501717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1502156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1503003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1503435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1503980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1504432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1505083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1505528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1506075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1506600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1507174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1507607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1508155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1508614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1509097Z [INFO] 2023-01-11 22:09:28,319 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1509564Z [INFO] 2023-01-11 22:09:28,331 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1510116Z [INFO] 2023-01-11 22:09:28,344 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1510559Z [INFO] 2023-01-11 22:09:28,365 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1511021Z [INFO] 2023-01-11 22:09:28,368 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1511499Z [INFO] 2023-01-11 22:09:28,368 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1511951Z [INFO] 2023-01-11 22:09:28,369 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1512420Z [INFO] 2023-01-11 22:09:28,372 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1512986Z [INFO] 2023-01-11 22:09:29,434 api: Reaping worker_id=[45601]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1513490Z [INFO] 2023-01-11 22:09:29,434 api: Successfully reaped worker=[45601]
2023-01-11T22:09:36.1514030Z [INFO] 2023-01-11 22:09:29,465 api: Reaping worker_id=[45599]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1514534Z [INFO] 2023-01-11 22:09:29,466 api: Successfully reaped worker=[45599]
2023-01-11T22:09:36.1515081Z [INFO] 2023-01-11 22:09:29,486 api: Reaping worker_id=[45595]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1515578Z [INFO] 2023-01-11 22:09:29,487 api: Successfully reaped worker=[45595]
2023-01-11T22:09:36.1516103Z [INFO] 2023-01-11 22:09:29,487 api: Reaping worker_id=[45597]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1516604Z [INFO] 2023-01-11 22:09:29,487 api: Successfully reaped worker=[45597]
2023-01-11T22:09:36.1517030Z [INFO] 2023-01-11 22:09:29,499 api: Stopping LocalTimerServer
2023-01-11T22:09:36.1517432Z [INFO] 2023-01-11 22:09:29,499 api: Stopping watchdog thread...
2023-01-11T22:09:36.1517716Z ok (4.780s)
2023-01-11T22:09:36.1518275Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2023-01-11 22:09:29,510 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2023-01-11T22:09:36.1518801Z [INFO] 2023-01-11 22:09:29,510 api: Starting watchdog thread...
2023-01-11T22:09:36.1519329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1520172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1520865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1521312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1521883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1522323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1522962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1523417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1523993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1524431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1524985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1525442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1526012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1526446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1526996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1527459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1528026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1528459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1529003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1529465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1530034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1530453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1531015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1531477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1532051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1532467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1533032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1533488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1534078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1534496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1535058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1535512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1535978Z [INFO] 2023-01-11 22:09:31,505 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1536455Z [INFO] 2023-01-11 22:09:31,513 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1536925Z [INFO] 2023-01-11 22:09:31,526 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1537397Z [INFO] 2023-01-11 22:09:31,527 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1537922Z [INFO] 2023-01-11 22:09:31,533 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1538383Z [INFO] 2023-01-11 22:09:31,546 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1538845Z [INFO] 2023-01-11 22:09:31,547 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1539294Z [INFO] 2023-01-11 22:09:31,550 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1539907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1540359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1540930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1541379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1541952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1542386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1542935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1543394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1543963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1544403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1544950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1545404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1545970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1546388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1546949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1547404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1548202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1548623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1549185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1549640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1550204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1550626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1551188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1551640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1552185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1552617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1553185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1553635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1554185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:36.1554698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:36.1555263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:36.1555720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:36.1556180Z [INFO] 2023-01-11 22:09:34,563 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1556657Z [INFO] 2023-01-11 22:09:34,582 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1557189Z [INFO] 2023-01-11 22:09:34,602 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1557657Z [INFO] 2023-01-11 22:09:34,616 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1558115Z [INFO] 2023-01-11 22:09:34,626 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1558579Z [INFO] 2023-01-11 22:09:34,645 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1559023Z [INFO] 2023-01-11 22:09:34,649 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1559653Z [INFO] 2023-01-11 22:09:34,650 api: Timer client configured to: LocalTimerClient
2023-01-11T22:09:36.1560230Z [INFO] 2023-01-11 22:09:35,671 api: Reaping worker_id=[46147]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1560735Z [INFO] 2023-01-11 22:09:35,672 api: Successfully reaped worker=[46147]
2023-01-11T22:09:36.1561307Z [INFO] 2023-01-11 22:09:35,692 api: Reaping worker_id=[46140]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1561810Z [INFO] 2023-01-11 22:09:35,692 api: Successfully reaped worker=[46140]
2023-01-11T22:09:36.1562364Z [INFO] 2023-01-11 22:09:35,723 api: Reaping worker_id=[46145]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1562852Z [INFO] 2023-01-11 22:09:35,723 api: Successfully reaped worker=[46145]
2023-01-11T22:09:36.1563380Z [INFO] 2023-01-11 22:09:35,733 api: Reaping worker_id=[46144]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1563885Z [INFO] 2023-01-11 22:09:35,733 api: Successfully reaped worker=[46144]
2023-01-11T22:09:36.1564428Z [INFO] 2023-01-11 22:09:35,744 api: Reaping worker_id=[46141]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1564950Z [INFO] 2023-01-11 22:09:35,744 local_timer: Process with pid=46141 does not exist. Skipping
2023-01-11T22:09:36.1565412Z [INFO] 2023-01-11 22:09:35,744 api: Successfully reaped worker=[46141]
2023-01-11T22:09:36.1565840Z [INFO] 2023-01-11 22:09:35,749 api: Stopping LocalTimerServer
2023-01-11T22:09:36.1566261Z [INFO] 2023-01-11 22:09:35,749 api: Stopping watchdog thread...
2023-01-11T22:09:36.1566784Z [INFO] 2023-01-11 22:09:35,754 api: Reaping worker_id=[46142]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:09:36.1567316Z [INFO] 2023-01-11 22:09:35,754 local_timer: Process with pid=46142 does not exist. Skipping
2023-01-11T22:09:36.1567779Z [INFO] 2023-01-11 22:09:35,754 api: Successfully reaped worker=[46142]
2023-01-11T22:09:36.1568056Z ok (6.246s)
2023-01-11T22:09:36.1568204Z 
2023-01-11T22:09:36.1568468Z ----------------------------------------------------------------------
2023-01-11T22:09:36.1568792Z Ran 2 tests in 11.026s
2023-01-11T22:09:36.1568954Z 
2023-01-11T22:09:36.1569045Z OK
2023-01-11T22:09:36.1569161Z 
2023-01-11T22:09:36.1569285Z Generating XML reports...
2023-01-11T22:09:36.1569911Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20230111220924.xml
2023-01-11T22:09:36.1570286Z 
2023-01-11T22:09:36.1570616Z ##[endgroup]
2023-01-11T22:09:36.1571257Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_z8xif160)
2023-01-11T22:09:36.1571646Z 
2023-01-11T22:09:36.1571949Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2023-01-11 22:09:36.148209]
2023-01-11T22:09:36.1572794Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:09:36.148460]
2023-01-11T22:09:49.6485697Z 
2023-01-11T22:09:49.6486446Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_binary_cmp
2023-01-11T22:09:49.6488686Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_sw5214aq)
2023-01-11T22:09:49.6489631Z 
2023-01-11T22:09:49.6489854Z Running tests...
2023-01-11T22:09:49.6490481Z ----------------------------------------------------------------------
2023-01-11T22:09:49.6491094Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2023-01-11T22:09:49.6491573Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2023-01-11T22:09:49.6492029Z Test torch.allclose(ShardedTensor, ShardedTensor) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:09:49.6492494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46447
2023-01-11T22:09:49.6492930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46448
2023-01-11T22:09:49.6493369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46449
2023-01-11T22:09:49.6493787Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46450
2023-01-11T22:09:49.6494422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6494881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6495444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6495894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6496471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6496911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6497492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6497929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6498505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6498941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6499490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6499954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6500526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6500959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6501593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6502041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6502485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:49.6502954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:49.6503405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:49.6503866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:49.6504248Z skip: Need at least 4 CUDA devices (3.952s)
2023-01-11T22:09:49.6504904Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46583
2023-01-11T22:09:49.6505429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46584
2023-01-11T22:09:49.6505873Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46585
2023-01-11T22:09:49.6506312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46586
2023-01-11T22:09:49.6506986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6507424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6508002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6508465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6509026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6509470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6510039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6510496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6511060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6511497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6512063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6512499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6513071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6513523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6514080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6514515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6514951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:49.6515424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:49.6515888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:49.6516334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:49.6516717Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:09:49.6517077Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2023-01-11T22:09:49.6517555Z Test torch.equal(ShardedTensor, ShardedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46719
2023-01-11T22:09:49.6518066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46720
2023-01-11T22:09:49.6518508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46721
2023-01-11T22:09:49.6518925Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46722
2023-01-11T22:09:49.6522396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6522859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6523442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6523891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6524599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6525038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6525609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6526050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6526688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6527135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6527686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6528156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6528714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6529150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6529699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6530151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6530578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:49.6531033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:49.6531497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:49.6531952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:49.6532339Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:09:49.6532833Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46855
2023-01-11T22:09:49.6533375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46856
2023-01-11T22:09:49.6533818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 46857
2023-01-11T22:09:49.6534256Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 46858
2023-01-11T22:09:49.6534850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6535293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6535863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6536311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6536894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6537328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6537892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6538333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6538904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6539342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6539892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6540347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6540914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:09:49.6541421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:09:49.6541973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:09:49.6542430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:09:49.6542862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:09:49.6543383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:09:49.6543839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:09:49.6544303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:09:49.6544681Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:09:49.6544872Z 
2023-01-11T22:09:49.6545131Z ----------------------------------------------------------------------
2023-01-11T22:09:49.6545478Z Ran 4 tests in 11.183s
2023-01-11T22:09:49.6545632Z 
2023-01-11T22:09:49.6545741Z OK (skipped=4)
2023-01-11T22:09:49.6545894Z 
2023-01-11T22:09:49.6546018Z Generating XML reports...
2023-01-11T22:09:49.6546675Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20230111220938.xml
2023-01-11T22:09:49.6547074Z 
2023-01-11T22:09:49.6547409Z ##[endgroup]
2023-01-11T22:09:49.6548092Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_sw5214aq)
2023-01-11T22:09:49.6548491Z 
2023-01-11T22:09:49.6548753Z Running distributed/_shard/test_partial_tensor ... [2023-01-11 22:09:49.648679]
2023-01-11T22:09:49.6549438Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:09:49.648938]
2023-01-11T22:10:05.5368238Z 
2023-01-11T22:10:05.5369077Z Expand the folded group to see the log file of distributed/_shard/test_partial_tensor
2023-01-11T22:10:05.5370599Z ##[group]PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_p0dfqasq)
2023-01-11T22:10:05.5370972Z 
2023-01-11T22:10:05.5371165Z Running tests...
2023-01-11T22:10:05.5371880Z ----------------------------------------------------------------------
2023-01-11T22:10:05.5372666Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2023-01-11T22:10:05.5373468Z   test_cat (__main__.TestPartialTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:10:05.5373905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47026
2023-01-11T22:10:05.5374357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47027
2023-01-11T22:10:05.5374810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47028
2023-01-11T22:10:05.5375239Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47029
2023-01-11T22:10:05.5375860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5376313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5376895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5377348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5377926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5378370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5378923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5379668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5380249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5380695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5381248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5381798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5382386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5382822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5383373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5383838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5384269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:05.5384721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:10:05.5385178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:10:05.5385648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:05.5386040Z skip: Need at least 4 CUDA devices (4.045s)
2023-01-11T22:10:05.5386491Z   test_cat_errors (__main__.TestPartialTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47162
2023-01-11T22:10:05.5387005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47163
2023-01-11T22:10:05.5387444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47164
2023-01-11T22:10:05.5387868Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47165
2023-01-11T22:10:05.5388469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5388916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5389539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5389993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5390569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5391012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5391561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5392025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5392596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5393032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5393581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5394040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5394613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5395050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5395600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5396130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5396560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:05.5397010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:05.5397468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:10:05.5397938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:10:05.5398373Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:10:05.5398832Z   test_transpose (__main__.TestPartialTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47298
2023-01-11T22:10:05.5399342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47299
2023-01-11T22:10:05.5400104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47300
2023-01-11T22:10:05.5400534Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47301
2023-01-11T22:10:05.5401154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5401598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5402168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5402615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5403189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5403629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5404179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5404638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5405214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5405653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5406207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5406665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5407243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5407679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5408227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5408684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5409120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:05.5409570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:10:05.5410028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:10:05.5410480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:05.5410866Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:10:05.5411350Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47434
2023-01-11T22:10:05.5411890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47435
2023-01-11T22:10:05.5412335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47436
2023-01-11T22:10:05.5412755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47437
2023-01-11T22:10:05.5413484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5413931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5414500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5414948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5415590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5416043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5416615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5417055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5417629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5418065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5418614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5419072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5419645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5420079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5420624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5421084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5421513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:10:05.5421970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:05.5422430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:10:05.5422884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:05.5423269Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:10:05.5423757Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47570
2023-01-11T22:10:05.5424302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47571
2023-01-11T22:10:05.5424744Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 47572
2023-01-11T22:10:05.5425180Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 47573
2023-01-11T22:10:05.5425775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5426219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5426783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5427228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5427805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5428247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5428812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5429254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5429822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5430329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5430882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5431338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5431906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:05.5432405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:05.5432968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:05.5433424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:05.5433856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:10:05.5434330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:05.5434776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:10:05.5435254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:05.5435632Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:10:05.5435825Z 
2023-01-11T22:10:05.5436080Z ----------------------------------------------------------------------
2023-01-11T22:10:05.5436411Z Ran 5 tests in 13.585s
2023-01-11T22:10:05.5436572Z 
2023-01-11T22:10:05.5436680Z OK (skipped=5)
2023-01-11T22:10:05.5436830Z 
2023-01-11T22:10:05.5436955Z Generating XML reports...
2023-01-11T22:10:05.5437554Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20230111220951.xml
2023-01-11T22:10:05.5438364Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20230111220951.xml
2023-01-11T22:10:05.5438739Z 
2023-01-11T22:10:05.5439070Z ##[endgroup]
2023-01-11T22:10:05.5442474Z FINISHED PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_p0dfqasq)
2023-01-11T22:10:05.5442844Z 
2023-01-11T22:10:05.5443130Z Running distributed/fsdp/test_fsdp_overlap ... [2023-01-11 22:10:05.537000]
2023-01-11T22:10:05.5443821Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:10:05.537307]
2023-01-11T22:10:21.8098088Z 
2023-01-11T22:10:21.8098824Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap
2023-01-11T22:10:21.8100605Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_eg8nts6g)
2023-01-11T22:10:21.8101148Z 
2023-01-11T22:10:21.8101269Z Running tests...
2023-01-11T22:10:21.8101800Z ----------------------------------------------------------------------
2023-01-11T22:10:21.8102357Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2023-01-11T22:10:21.8102910Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:10:21.8104005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47741
2023-01-11T22:10:21.8105189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:21.8105991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:21.8107079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:21.8107968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:21.8109295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:10:21.8110649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:10:21.8111716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:21.8113431Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:10:21.8114407Z   warnings.warn(
2023-01-11T22:10:21.8116757Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:10:21.8118316Z   warnings.warn(
2023-01-11T22:10:21.8118809Z dist init r=0, world=1
2023-01-11T22:10:21.8119083Z 
2023-01-11T22:10:21.8119273Z rank0:
2023-01-11T22:10:21.8120683Z   e1: {'cpu_iter': 0.0017847089000000871, 'cpu_wait': 3.592700000005777e-05, 'gpu_compute': 0.06315200002864003, 'gpu_total': 0.7415680050849914}
2023-01-11T22:10:21.8121863Z   e2: {'cpu_iter': 0.005673516799999856, 'cpu_wait': 3.462550000010722e-05, 'gpu_compute': 0.25248959809541704, 'gpu_total': 2.3607871770858764}
2023-01-11T22:10:21.8123028Z   e3: {'cpu_iter': 0.0019034596000000903, 'cpu_wait': 0.18845791740000006, 'gpu_compute': 190.7001609802246, 'gpu_total': 190.98700103759765}
2023-01-11T22:10:21.8124133Z   e4: {'cpu_iter': 0.00575218310000043, 'cpu_wait': 0.1855744590999997, 'gpu_compute': 190.72147216796876, 'gpu_total': 191.2682876586914}
2023-01-11T22:10:21.8124765Z ok (13.957s)
2023-01-11T22:10:21.8126890Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T22:10:21.8128153Z 
2023-01-11T22:10:21.8128684Z ----------------------------------------------------------------------
2023-01-11T22:10:21.8129296Z Ran 2 tests in 13.957s
2023-01-11T22:10:21.8129614Z 
2023-01-11T22:10:21.8129821Z OK (skipped=1)
2023-01-11T22:10:21.8130134Z 
2023-01-11T22:10:21.8130369Z Generating XML reports...
2023-01-11T22:10:21.8131677Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20230111221007.xml
2023-01-11T22:10:21.8133452Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20230111221007.xml
2023-01-11T22:10:21.8134279Z 
2023-01-11T22:10:21.8134890Z ##[endgroup]
2023-01-11T22:10:21.8136092Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_eg8nts6g)
2023-01-11T22:10:21.8136789Z 
2023-01-11T22:10:21.8137394Z Running distributed/tensor/parallel/test_tp_examples ... [2023-01-11 22:10:21.809790]
2023-01-11T22:10:21.8138903Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:10:21.810115]
2023-01-11T22:10:39.0701801Z 
2023-01-11T22:10:39.0702587Z Expand the folded group to see the log file of distributed/tensor/parallel/test_tp_examples
2023-01-11T22:10:39.0703568Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_examples_pl7fkoci)
2023-01-11T22:10:39.0704290Z 
2023-01-11T22:10:39.0704416Z Running tests...
2023-01-11T22:10:39.0704922Z ----------------------------------------------------------------------
2023-01-11T22:10:39.0705523Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples
2023-01-11T22:10:39.0706071Z   test_mlp_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:10:39.0706584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47818
2023-01-11T22:10:39.0707125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47819
2023-01-11T22:10:39.0707775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:39.0708194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:39.0708778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:39.0709245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:39.0709830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:39.0710271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:39.0710835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:39.0711291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:39.0711712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:39.0712201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:10:39.0712684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:39.0713168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:10:39.0713804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:39.0714487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:39.0714875Z ok (6.020s)
2023-01-11T22:10:39.0715370Z   test_self_attn_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47903
2023-01-11T22:10:39.0715906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47904
2023-01-11T22:10:39.0716508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:39.0716950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:39.0717524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:39.0717978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:39.0719992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:39.0720449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:39.0721007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:39.0721479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:39.0721916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:39.0722401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:10:39.0722866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:39.0723479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:10:39.0724147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:39.0724830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:39.0725202Z ok (4.514s)
2023-01-11T22:10:39.0725752Z   test_self_attn_replacement_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47988
2023-01-11T22:10:39.0726339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47989
2023-01-11T22:10:39.0726935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:39.0727382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:39.0727956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:39.0728464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:39.0729024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:39.0729466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:39.0730036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:39.0730494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:39.0730909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:39.0731393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:10:39.0731881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:39.0732342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:10:39.0732994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:39.0733680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:39.0734069Z ok (4.414s)
2023-01-11T22:10:39.0734202Z 
2023-01-11T22:10:39.0734471Z ----------------------------------------------------------------------
2023-01-11T22:10:39.0734798Z Ran 3 tests in 14.949s
2023-01-11T22:10:39.0734958Z 
2023-01-11T22:10:39.0735051Z OK
2023-01-11T22:10:39.0735181Z 
2023-01-11T22:10:39.0735287Z Generating XML reports...
2023-01-11T22:10:39.0735968Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20230111221023.xml
2023-01-11T22:10:39.0736389Z 
2023-01-11T22:10:39.0736719Z ##[endgroup]
2023-01-11T22:10:39.0737366Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_examples_pl7fkoci)
2023-01-11T22:10:39.0737766Z 
2023-01-11T22:10:39.0738035Z Running distributed/fsdp/test_fsdp_memory ... [2023-01-11 22:10:39.070251]
2023-01-11T22:10:39.0738719Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:10:39.070569]
2023-01-11T22:10:58.1465976Z 
2023-01-11T22:10:58.1466740Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory
2023-01-11T22:10:58.1468183Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_s1t6rtmm)
2023-01-11T22:10:58.1468848Z 
2023-01-11T22:10:58.1468959Z Running tests...
2023-01-11T22:10:58.1469474Z ----------------------------------------------------------------------
2023-01-11T22:10:58.1470031Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2023-01-11T22:10:58.1470542Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:10:58.1471012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48108
2023-01-11T22:10:58.1471565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48109
2023-01-11T22:10:58.1472194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:58.1472621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:58.1473194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:58.1473669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:58.1474229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:58.1474670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:58.1475239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:58.1475686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:58.1476137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:10:58.1476633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:10:58.1477291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:58.1477966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:58.1478482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:58.1478950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:58.1480523Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:10:58.1481298Z   warnings.warn(
2023-01-11T22:10:58.1482443Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:10:58.1483220Z   warnings.warn(
2023-01-11T22:10:58.1483454Z dist init r=0, world=2
2023-01-11T22:10:58.1483703Z dist init r=1, world=2
2023-01-11T22:10:58.1483921Z ok (9.347s)
2023-01-11T22:10:58.1484350Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48221
2023-01-11T22:10:58.1484871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48222
2023-01-11T22:10:58.1485461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:58.1486024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:58.1486601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:58.1487065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:58.1487624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:10:58.1488065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:10:58.1488705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:10:58.1489207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:10:58.1489662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:10:58.1490133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:10:58.1490798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:58.1491482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:10:58.1491982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:10:58.1492449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:10:58.1493719Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:10:58.1494510Z   warnings.warn(
2023-01-11T22:10:58.1495641Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:10:58.1496391Z   warnings.warn(
2023-01-11T22:10:58.1496639Z dist init r=1, world=2
2023-01-11T22:10:58.1496889Z dist init r=0, world=2
2023-01-11T22:10:58.1497108Z ok (7.417s)
2023-01-11T22:10:58.1497254Z 
2023-01-11T22:10:58.1497527Z ----------------------------------------------------------------------
2023-01-11T22:10:58.1497855Z Ran 2 tests in 16.764s
2023-01-11T22:10:58.1498014Z 
2023-01-11T22:10:58.1498107Z OK
2023-01-11T22:10:58.1498254Z 
2023-01-11T22:10:58.1498356Z Generating XML reports...
2023-01-11T22:10:58.1498930Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20230111221040.xml
2023-01-11T22:10:58.1499276Z 
2023-01-11T22:10:58.1499603Z ##[endgroup]
2023-01-11T22:10:58.1500179Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_s1t6rtmm)
2023-01-11T22:10:58.1500527Z 
2023-01-11T22:10:58.1500833Z Running distributed/test_c10d_object_collectives ... [2023-01-11 22:10:58.146634]
2023-01-11T22:10:58.1501507Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:10:58.146924]
2023-01-11T22:11:17.6830566Z 
2023-01-11T22:11:17.6831079Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives
2023-01-11T22:11:17.6832356Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_y6tjxm9e)
2023-01-11T22:11:17.6833626Z 
2023-01-11T22:11:17.6833865Z Running tests...
2023-01-11T22:11:17.6834852Z ----------------------------------------------------------------------
2023-01-11T22:11:17.6836035Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives
2023-01-11T22:11:17.6836798Z   test_all_gather_object (__main__.TestObjectCollectives) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:11:17.6837404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48369
2023-01-11T22:11:17.6837871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48370
2023-01-11T22:11:17.6838496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6838947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6839741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6840208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6840792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6841239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6841818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6842276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6842708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:17.6843180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:17.6843657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:17.6844141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:17.6844800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6845487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6845862Z ok (5.525s)
2023-01-11T22:11:17.6846308Z   test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48448
2023-01-11T22:11:17.6846838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48449
2023-01-11T22:11:17.6847446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6847881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6848454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6848922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6849482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6849923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6850493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6850942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6851389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:17.6851880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:17.6852471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:17.6852930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:17.6853586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6854301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6854769Z ok (3.910s)
2023-01-11T22:11:17.6855193Z   test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48527
2023-01-11T22:11:17.6855717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48528
2023-01-11T22:11:17.6856327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6856781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6857343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6857808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6858382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6858807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6859376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6859835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6860268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:17.6860741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:17.6861221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:17.6861708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:17.6862361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6863025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6863416Z ok (3.910s)
2023-01-11T22:11:17.6863850Z   test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48608
2023-01-11T22:11:17.6864360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48609
2023-01-11T22:11:17.6864967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6865420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6865989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6866434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6867009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:17.6867454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:17.6868022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:17.6868464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:17.6868895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:17.6869452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:17.6869960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:17.6870438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:17.6871096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6871840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:17.6872214Z ok (3.910s)
2023-01-11T22:11:17.6872362Z 
2023-01-11T22:11:17.6872629Z ----------------------------------------------------------------------
2023-01-11T22:11:17.6872959Z Ran 4 tests in 17.256s
2023-01-11T22:11:17.6873124Z 
2023-01-11T22:11:17.6873193Z OK
2023-01-11T22:11:17.6873330Z 
2023-01-11T22:11:17.6873455Z Generating XML reports...
2023-01-11T22:11:17.6874071Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20230111221100.xml
2023-01-11T22:11:17.6874433Z 
2023-01-11T22:11:17.6874740Z ##[endgroup]
2023-01-11T22:11:17.6875362Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_y6tjxm9e)
2023-01-11T22:11:17.6875721Z 
2023-01-11T22:11:17.6876031Z Running distributed/checkpoint/test_file_system_checkpoint ... [2023-01-11 22:11:17.683038]
2023-01-11T22:11:17.6876768Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:11:17.683285]
2023-01-11T22:11:43.0821667Z 
2023-01-11T22:11:43.0822173Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint
2023-01-11T22:11:43.0824692Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_12vk804w)
2023-01-11T22:11:43.0825130Z 
2023-01-11T22:11:43.0825223Z Running tests...
2023-01-11T22:11:43.0827082Z ----------------------------------------------------------------------
2023-01-11T22:11:43.0827719Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint
2023-01-11T22:11:43.0828329Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:11:43.0828824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48724
2023-01-11T22:11:43.0830889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48725
2023-01-11T22:11:43.0831597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0832081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0832660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0833125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0833709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0834158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0834742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0835193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0836101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:43.0836609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:43.0837345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:43.0837850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:43.0838539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0839329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0840335Z ok (5.645s)
2023-01-11T22:11:43.0840799Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48807
2023-01-11T22:11:43.0841354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48808
2023-01-11T22:11:43.0841978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0842433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0842988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0843456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0844030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0844458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0845030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0845490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0845935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:43.0846380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:43.0846856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:43.0847345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:43.0848001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0848671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0849064Z ok (4.613s)
2023-01-11T22:11:43.0849510Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48890
2023-01-11T22:11:43.0850033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48891
2023-01-11T22:11:43.0850640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0851083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0851650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0852103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0852680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0853125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0853674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0854134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0854567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:43.0855159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:43.0855625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:43.0856114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:43.0856770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0857519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0857907Z ok (4.011s)
2023-01-11T22:11:43.0858395Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48973
2023-01-11T22:11:43.0858960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48974
2023-01-11T22:11:43.0859554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0859997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0860567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0861029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0861590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0862033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0862600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0863054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0863470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:43.0863935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:43.0864414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:43.0864887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:43.0865542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0866222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0866610Z ok (4.713s)
2023-01-11T22:11:43.0867818Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:11:43.0868733Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:11:43.0868990Z ok (0.049s)
2023-01-11T22:11:43.0869513Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49056
2023-01-11T22:11:43.0870118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49057
2023-01-11T22:11:43.0870712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0871156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0871812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0872311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0872869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:11:43.0873306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:11:43.0873922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:11:43.0874386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:11:43.0874799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:11:43.0875271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:11:43.0875752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:11:43.0876228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:11:43.0876878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0877559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:11:43.0878727Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:11:43.0879745Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:11:43.0880010Z ok (4.011s)
2023-01-11T22:11:43.0880162Z 
2023-01-11T22:11:43.0880446Z ----------------------------------------------------------------------
2023-01-11T22:11:43.0880761Z Ran 6 tests in 23.042s
2023-01-11T22:11:43.0880921Z 
2023-01-11T22:11:43.0880996Z OK
2023-01-11T22:11:43.0881127Z 
2023-01-11T22:11:43.0881250Z Generating XML reports...
2023-01-11T22:11:43.0881931Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20230111221119.xml
2023-01-11T22:11:43.0882865Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20230111221119.xml
2023-01-11T22:11:43.0883851Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111221119.xml
2023-01-11T22:11:43.0884339Z 
2023-01-11T22:11:43.0884721Z ##[endgroup]
2023-01-11T22:11:43.0885402Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_12vk804w)
2023-01-11T22:11:43.0885807Z 
2023-01-11T22:11:43.0886104Z Running distributed/elastic/multiprocessing/api_test ... [2023-01-11 22:11:43.082193]
2023-01-11T22:11:43.0886806Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:11:43.082447]
2023-01-11T22:12:12.7358607Z 
2023-01-11T22:12:12.7359374Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test
2023-01-11T22:12:12.7360684Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_51kvzf7_)
2023-01-11T22:12:12.7361084Z 
2023-01-11T22:12:12.7361175Z Running tests...
2023-01-11T22:12:12.7367922Z ----------------------------------------------------------------------
2023-01-11T22:12:12.7369495Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2023-01-11T22:12:12.7369987Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.644s)
2023-01-11T22:12:12.7370370Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2023-01-11T22:12:12.7370773Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T22:12:12.7371208Z   test_binary (__main__.StartProcessesListTest) ... hello stderr from 1
2023-01-11T22:12:12.7371837Z hello stdout from 1
2023-01-11T22:12:12.7372224Z hello stderr from 0
2023-01-11T22:12:12.7372560Z hello stdout from 0
2023-01-11T22:12:12.7372781Z ok (0.104s)
2023-01-11T22:12:12.7373110Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1
2023-01-11T22:12:12.7373434Z bar stdout from 1
2023-01-11T22:12:12.7373847Z failed (exitcode: 138) local_rank: 0 (pid: 49175) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2023-01-11T22:12:12.7374256Z ok (0.105s)
2023-01-11T22:12:12.7374653Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T22:12:12.7375123Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2023-01-11T22:12:12.7375608Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2023-01-11T22:12:12.7376227Z     raise RuntimeError(f"raised from {rank}")
2023-01-11T22:12:12.7376791Z RuntimeError: raised from 0
2023-01-11T22:12:12.7377196Z bar from 1
2023-01-11T22:12:12.7377640Z failed (exitcode: 1) local_rank: 0 (pid: 49178) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2023-01-11T22:12:12.7378035Z ok (0.104s)
2023-01-11T22:12:12.7378365Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2023-01-11T22:12:12.7378736Z [trainer1]:world stderr from 1
2023-01-11T22:12:12.7379021Z [trainer0]:hello stdout from 0
2023-01-11T22:12:12.7379269Z ok (1.106s)
2023-01-11T22:12:12.7379907Z   test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7380419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7380992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7381446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7382021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7382469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7383037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7383487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7383806Z hello stderr from 0
2023-01-11T22:12:12.7384059Z hello stderr from 1
2023-01-11T22:12:12.7384399Z hello stdout from 0
2023-01-11T22:12:12.7384644Z hello stdout from 1
2023-01-11T22:12:12.7384904Z Closing process 49185 via signal SIGTERM
2023-01-11T22:12:12.7385170Z ok (2.321s)
2023-01-11T22:12:12.7385818Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7386332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7386902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7387368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7387945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7388467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7389036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7389497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7390047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7390541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7391120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7391578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7392133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7392578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7393142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7393598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7393923Z Closing process 49251 via signal SIGTERM
2023-01-11T22:12:12.7394228Z Closing process 49252 via signal SIGTERM
2023-01-11T22:12:12.7394528Z Closing process 49254 via signal SIGTERM
2023-01-11T22:12:12.7394782Z ok (2.784s)
2023-01-11T22:12:12.7395085Z   test_function_raise (__main__.StartProcessesListTest)
2023-01-11T22:12:12.7395775Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7396265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7396842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7397307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7397879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7398303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7398873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7399335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7400195Z failed (exitcode: 1) local_rank: 0 (pid: 49383) of fn: echo2 (start_method: spawn)
2023-01-11T22:12:12.7400549Z Traceback (most recent call last):
2023-01-11T22:12:12.7401095Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:12:12.7401528Z     self._pc.join(-1)
2023-01-11T22:12:12.7401982Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2023-01-11T22:12:12.7402443Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2023-01-11T22:12:12.7402898Z torch.multiprocessing.spawn.ProcessRaisedException: 
2023-01-11T22:12:12.7403168Z 
2023-01-11T22:12:12.7403378Z -- Process 0 terminated with the following error:
2023-01-11T22:12:12.7403701Z Traceback (most recent call last):
2023-01-11T22:12:12.7404197Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2023-01-11T22:12:12.7404551Z     fn(i, *args)
2023-01-11T22:12:12.7405036Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2023-01-11T22:12:12.7405426Z     ret = record(fn)(*args_)
2023-01-11T22:12:12.7405962Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2023-01-11T22:12:12.7406474Z     return f(*args, **kwargs)
2023-01-11T22:12:12.7406868Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2023-01-11T22:12:12.7407246Z     raise RuntimeError(msg)
2023-01-11T22:12:12.7407493Z RuntimeError: hello
2023-01-11T22:12:12.7407654Z 
2023-01-11T22:12:12.7407754Z ok (2.322s)
2023-01-11T22:12:12.7408085Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T22:12:12.7408567Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T22:12:12.7409032Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 49449 via signal SIGTERM
2023-01-11T22:12:12.7409394Z ok (0.003s)
2023-01-11T22:12:12.7410005Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2023-01-11T22:12:12.7410459Z Traceback (most recent call last):
2023-01-11T22:12:12.7410987Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:12:12.7411407Z     self._pc.join(-1)
2023-01-11T22:12:12.7411737Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2023-01-11T22:12:12.7412067Z     return self._mock_call(*args, **kwargs)
2023-01-11T22:12:12.7412424Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2023-01-11T22:12:12.7412788Z     return self._execute_mock_call(*args, **kwargs)
2023-01-11T22:12:12.7413147Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2023-01-11T22:12:12.7413472Z     raise effect
2023-01-11T22:12:12.7413840Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2023-01-11T22:12:12.7414179Z ok (0.004s)
2023-01-11T22:12:12.7414804Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7415320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7415895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7416346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7416651Z ok (3.283s)
2023-01-11T22:12:12.7417039Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 49483 closing signal SIGTERM
2023-01-11T22:12:12.7417391Z ok (0.005s)
2023-01-11T22:12:12.7417700Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T22:12:12.7418096Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T22:12:12.7418806Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7419307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7419875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7420338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7420897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7421343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7421912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7422373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7422655Z world
2023-01-11T22:12:12.7422864Z hello
2023-01-11T22:12:12.7423120Z Closing process 49484 via signal SIGTERM
2023-01-11T22:12:12.7423451Z ok (2.423s)
2023-01-11T22:12:12.7423771Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T22:12:12.7424172Z   test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1
2023-01-11T22:12:12.7424470Z bar stdout from 1
2023-01-11T22:12:12.7424893Z failed (exitcode: 138) local_rank: 0 (pid: 49550) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2023-01-11T22:12:12.7425279Z ok (0.104s)
2023-01-11T22:12:12.7425648Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T22:12:12.7426092Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2023-01-11T22:12:12.7426571Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2023-01-11T22:12:12.7426982Z     raise RuntimeError(f"raised from {rank}")
2023-01-11T22:12:12.7427262Z RuntimeError: raised from 0
2023-01-11T22:12:12.7427522Z bar from 1
2023-01-11T22:12:12.7427934Z failed (exitcode: 1) local_rank: 0 (pid: 49553) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2023-01-11T22:12:12.7428306Z ok (0.104s)
2023-01-11T22:12:12.7428941Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7429461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7430034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7430485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7431053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7431493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7432048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7432509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7433079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7433516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7434067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7434524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7435094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7435529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7436081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7436537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7436880Z Closing process 49556 via signal SIGTERM
2023-01-11T22:12:12.7437168Z Closing process 49557 via signal SIGTERM
2023-01-11T22:12:12.7437469Z Closing process 49558 via signal SIGTERM
2023-01-11T22:12:12.7437733Z ok (2.777s)
2023-01-11T22:12:12.7438008Z   test_function_raise (__main__.StartProcessesTest)
2023-01-11T22:12:12.7438686Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7439190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7440249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7440841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7441420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7441857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7442405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7442866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7443337Z failed (exitcode: 1) local_rank: 0 (pid: 49687) of fn: echo2 (start_method: spawn)
2023-01-11T22:12:12.7443696Z Traceback (most recent call last):
2023-01-11T22:12:12.7444216Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:12:12.7444641Z     self._pc.join(-1)
2023-01-11T22:12:12.7445109Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2023-01-11T22:12:12.7445555Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2023-01-11T22:12:12.7446008Z torch.multiprocessing.spawn.ProcessRaisedException: 
2023-01-11T22:12:12.7446276Z 
2023-01-11T22:12:12.7446501Z -- Process 0 terminated with the following error:
2023-01-11T22:12:12.7446821Z Traceback (most recent call last):
2023-01-11T22:12:12.7447296Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2023-01-11T22:12:12.7447648Z     fn(i, *args)
2023-01-11T22:12:12.7448149Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2023-01-11T22:12:12.7448521Z     ret = record(fn)(*args_)
2023-01-11T22:12:12.7449058Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2023-01-11T22:12:12.7449462Z     return f(*args, **kwargs)
2023-01-11T22:12:12.7449858Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2023-01-11T22:12:12.7450215Z     raise RuntimeError(msg)
2023-01-11T22:12:12.7450482Z RuntimeError: hello
2023-01-11T22:12:12.7450643Z 
2023-01-11T22:12:12.7450745Z ok (2.322s)
2023-01-11T22:12:12.7451047Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T22:12:12.7451444Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T22:12:12.7451896Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 49753 via signal SIGTERM
2023-01-11T22:12:12.7452230Z ok (0.003s)
2023-01-11T22:12:12.7452827Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2023-01-11T22:12:12.7453290Z Traceback (most recent call last):
2023-01-11T22:12:12.7453820Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:12:12.7454235Z     self._pc.join(-1)
2023-01-11T22:12:12.7454565Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2023-01-11T22:12:12.7454912Z     return self._mock_call(*args, **kwargs)
2023-01-11T22:12:12.7455246Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2023-01-11T22:12:12.7455605Z     return self._execute_mock_call(*args, **kwargs)
2023-01-11T22:12:12.7455981Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2023-01-11T22:12:12.7456287Z     raise effect
2023-01-11T22:12:12.7456663Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2023-01-11T22:12:12.7457018Z ok (0.005s)
2023-01-11T22:12:12.7457626Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7458115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7458779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7459245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7459533Z ok (3.265s)
2023-01-11T22:12:12.7459911Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 49787 closing signal SIGTERM
2023-01-11T22:12:12.7460269Z ok (0.003s)
2023-01-11T22:12:12.7460623Z   test_to_map (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T22:12:12.7460998Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T22:12:12.7461691Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7462197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7462757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7463223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7463794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:12.7464238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:12.7464793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:12.7465251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:12.7465550Z world
2023-01-11T22:12:12.7465741Z hello
2023-01-11T22:12:12.7466000Z Closing process 49788 via signal SIGTERM
2023-01-11T22:12:12.7466267Z ok (2.418s)
2023-01-11T22:12:12.7466542Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2023-01-11T22:12:12.7466878Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2023-01-11T22:12:12.7467204Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2023-01-11T22:12:12.7467402Z 
2023-01-11T22:12:12.7467680Z ----------------------------------------------------------------------
2023-01-11T22:12:12.7467993Z Ran 38 tests in 27.237s
2023-01-11T22:12:12.7468154Z 
2023-01-11T22:12:12.7468248Z OK
2023-01-11T22:12:12.7468380Z 
2023-01-11T22:12:12.7468503Z Generating XML reports...
2023-01-11T22:12:12.7469137Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20230111221145.xml
2023-01-11T22:12:12.7470007Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20230111221145.xml
2023-01-11T22:12:12.7470853Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20230111221145.xml
2023-01-11T22:12:12.7471650Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20230111221145.xml
2023-01-11T22:12:12.7472006Z 
2023-01-11T22:12:12.7472344Z ##[endgroup]
2023-01-11T22:12:12.7473011Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_51kvzf7_)
2023-01-11T22:12:12.7473415Z 
2023-01-11T22:12:12.7473685Z Running distributed/_tensor/test_common_rules ... [2023-01-11 22:12:12.736050]
2023-01-11T22:12:12.7474347Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:12:12.736306]
2023-01-11T22:12:43.1819206Z 
2023-01-11T22:12:43.1819715Z Expand the folded group to see the log file of distributed/_tensor/test_common_rules
2023-01-11T22:12:43.1823846Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_h1_xeoew)
2023-01-11T22:12:43.1824737Z 
2023-01-11T22:12:43.1824925Z Running tests...
2023-01-11T22:12:43.1825477Z ----------------------------------------------------------------------
2023-01-11T22:12:43.1826022Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_common_rules
2023-01-11T22:12:43.1826535Z   test_einop_basic_propagation (__main__.CommonRulesTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:12:43.1829406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49889
2023-01-11T22:12:43.1830089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49890
2023-01-11T22:12:43.1830578Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 49891
2023-01-11T22:12:43.1831027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 49892
2023-01-11T22:12:43.1831703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1832157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1832744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1833220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1833784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1834233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1834818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1835284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1835841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1836305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1836875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1837340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1837895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1838358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1838932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1839376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1840535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1841027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1841499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1841945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1842331Z skip: Need at least 4 CUDA devices (4.033s)
2023-01-11T22:12:43.1842789Z   test_einop_errors (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50025
2023-01-11T22:12:43.1843291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50026
2023-01-11T22:12:43.1843746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50027
2023-01-11T22:12:43.1844198Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50028
2023-01-11T22:12:43.1844813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1845300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1846019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1846484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1847065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1847496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1848130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1848605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1849174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1849600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1850176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1850632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1851206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1851630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1852204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1852663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1853081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1853554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1854030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1854491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1854861Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:12:43.1855315Z   test_einop_linearity (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50161
2023-01-11T22:12:43.1855821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50162
2023-01-11T22:12:43.1856250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50163
2023-01-11T22:12:43.1856693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50164
2023-01-11T22:12:43.1857299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1857747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1858307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1858771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1859351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1859773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1860344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1860807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1861377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1861798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1862368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1862897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1863475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1863896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1864460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1864976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1865402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1865874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1866329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1866800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1867172Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:12:43.1867639Z   test_einop_merge_sharding (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50297
2023-01-11T22:12:43.1868160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50298
2023-01-11T22:12:43.1868693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50299
2023-01-11T22:12:43.1869139Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50300
2023-01-11T22:12:43.1869730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1870181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1870782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1871554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1872128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1872576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1873143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1873609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1874165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1874609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1875171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1875616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1876194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1876633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1877247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1877692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1878135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1878611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1879057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1879871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1880358Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:12:43.1880838Z   test_einop_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50433
2023-01-11T22:12:43.1881348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50434
2023-01-11T22:12:43.1881876Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50435
2023-01-11T22:12:43.1882863Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50436
2023-01-11T22:12:43.1883498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1883949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1884523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1884997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1885557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1886001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1886559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1887001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1887555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1888018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1888601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1889045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1889623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1890066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1890628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1891067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1891506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1891977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1892422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1892888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1893277Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:12:43.1893757Z   test_einop_pointwise_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50569
2023-01-11T22:12:43.1894270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50570
2023-01-11T22:12:43.1894711Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50571
2023-01-11T22:12:43.1895151Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50572
2023-01-11T22:12:43.1895763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1896189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1896750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1897195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1897823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1898288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1898871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1899330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1899935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1900386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1900961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1901403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1901985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1902431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1902999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1903440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1903877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1904350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1904817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1905263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1905646Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:12:43.1906160Z   test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50705
2023-01-11T22:12:43.1906695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50706
2023-01-11T22:12:43.1907136Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50707
2023-01-11T22:12:43.1907574Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50708
2023-01-11T22:12:43.1908184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1908615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1909188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1909657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1910223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1910662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1911238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1911698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1912257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1912695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1913264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1913722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1914279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1914780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1915353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1915795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1916230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1916748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1917229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1917676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1918058Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:12:43.1918548Z   test_pointwise_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50841
2023-01-11T22:12:43.1919060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50842
2023-01-11T22:12:43.1919844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50843
2023-01-11T22:12:43.1920290Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50844
2023-01-11T22:12:43.1920908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1921340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1921909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1922372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1922928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1923375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1923945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1924405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1924963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1925409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1925974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1926434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1926990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1927432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1928002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1928446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1928883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1929355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1929822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1930266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1930649Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:12:43.1931125Z   test_pointwise_rules_broadcasting (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50977
2023-01-11T22:12:43.1931721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50978
2023-01-11T22:12:43.1932166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 50979
2023-01-11T22:12:43.1932606Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 50980
2023-01-11T22:12:43.1933224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1933733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1934325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1934794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1935369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1935801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1936369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1936830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1937384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1937830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1938401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1938860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1939414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1939865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1940432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1940874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1941309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1941779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1942250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1942696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1943080Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:12:43.1943553Z   test_pointwise_rules_suggestion (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51113
2023-01-11T22:12:43.1944085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51114
2023-01-11T22:12:43.1944508Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51115
2023-01-11T22:12:43.1944940Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51116
2023-01-11T22:12:43.1945547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1945974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1946555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1947022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1947603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1948086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1948662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1949122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1949680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1950119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1950734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1951200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1951757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1952202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1952777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1953237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1953651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1954122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1954585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1955024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1955408Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:12:43.1955869Z   test_reduction_rule (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51249
2023-01-11T22:12:43.1956372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51250
2023-01-11T22:12:43.1956802Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 51251
2023-01-11T22:12:43.1957232Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 51252
2023-01-11T22:12:43.1957838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1958266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1958830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1959269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1960214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1960663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1961251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1961713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1962267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1962707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1963280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1963744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1964295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:12:43.1964736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:12:43.1965303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:12:43.1965849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:12:43.1966265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:12:43.1966733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:12:43.1967199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:12:43.1967701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:12:43.1968090Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:12:43.1968283Z 
2023-01-11T22:12:43.1968557Z ----------------------------------------------------------------------
2023-01-11T22:12:43.1968886Z Ran 11 tests in 28.137s
2023-01-11T22:12:43.1969030Z 
2023-01-11T22:12:43.1969140Z OK (skipped=11)
2023-01-11T22:12:43.1969300Z 
2023-01-11T22:12:43.1969424Z Generating XML reports...
2023-01-11T22:12:43.1970016Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20230111221214.xml
2023-01-11T22:12:43.1970359Z 
2023-01-11T22:12:43.1970801Z ##[endgroup]
2023-01-11T22:12:43.1971415Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_h1_xeoew)
2023-01-11T22:12:43.1971763Z 
2023-01-11T22:12:43.1972052Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2023-01-11 22:12:43.182380]
2023-01-11T22:12:43.1972756Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:12:43.182699]
2023-01-11T22:13:18.0129861Z 
2023-01-11T22:13:18.0130696Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T22:13:18.0131733Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_mpnp2inh)
2023-01-11T22:13:18.0132138Z 
2023-01-11T22:13:18.0132250Z Running tests...
2023-01-11T22:13:18.0138097Z ----------------------------------------------------------------------
2023-01-11T22:13:18.0138741Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2023-01-11T22:13:18.0139183Z   test_ddp_parity (__main__.TestClipGradNorm)
2023-01-11T22:13:18.0139669Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:13:18.0140159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51420
2023-01-11T22:13:18.0140837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51421
2023-01-11T22:13:18.0141995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:18.0143016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:18.0144125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:18.0144944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:18.0146026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:18.0147048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:18.0147846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:18.0148312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:18.0148770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:18.0149564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:18.0150868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:18.0151578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:18.0152084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:18.0152663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:18.0153147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0153633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0154651Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0155990Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0157233Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0158440Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0159173Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0160353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0161328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0162260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0164275Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0166484Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0167746Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0168974Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0170194Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0171653Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0172880Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0174098Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0174834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0175313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0176299Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0177518Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0178741Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0179956Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0181238Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0182439Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0183654Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0184870Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0186156Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0187411Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0188635Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0189856Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0191069Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0192269Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0193484Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0194704Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0195915Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0197125Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0198340Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0199986Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0201026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0201618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0202718Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0203962Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0205189Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0206412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0207136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0207602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0208590Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0209819Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0210542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0211022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0211996Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0213224Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0214445Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0215669Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0216465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0216930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0217973Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0219207Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0220426Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0221644Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0222866Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0224083Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0225298Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0226492Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0227712Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0228928Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0229644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0230125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0231110Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0232457Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0233698Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0234921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0235646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0236129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0237107Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0238328Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0239995Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0241240Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0242457Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0243669Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0244892Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0246102Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0247441Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0248702Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0249932Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0251157Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0252371Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0253579Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0254303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0254926Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0255910Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0257130Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0257850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0258328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0259565Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:13:18.0260369Z   return torch._VF.split_with_sizes(self, split_size, dim)
2023-01-11T22:13:18.0261548Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:13:18.0262425Z   return torch._VF.split_with_sizes(self, split_size, dim)
2023-01-11T22:13:18.0262852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0263335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0264371Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0265617Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0266844Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0268068Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0269281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0270504Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0271723Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0272939Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0273646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0274128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0274602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0275078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0276046Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0277339Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0278601Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0280332Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0281571Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0282790Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0284002Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0285209Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0285931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0286413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0287399Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0288605Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0289823Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0291017Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0292236Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0293559Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0294835Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0296068Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0297293Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0298512Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0299231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0299698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0300684Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0301904Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0302623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0303087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0304075Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0304785Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0305261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0306241Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0307447Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0308730Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0309505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0309996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0310985Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0312211Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0313420Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0314634Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0315359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0315840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0316831Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0318049Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0319255Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0320925Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0321653Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0322131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0323117Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0324428Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0325706Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0326941Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0328154Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0329365Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0330577Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0332069Z /opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:4772: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:13:18.0332962Z   return (linear(q, w_q, b_q),) + linear(k, w_kv, b_kv).chunk(2, dim=-1)
2023-01-11T22:13:18.0333378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0333857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0334854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0336085Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0337300Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0338594Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0339863Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0341095Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0342314Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0343539Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0344752Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0345972Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0347184Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0348404Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0349621Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0350812Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0352024Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0353307Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0354561Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0355774Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0356988Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0358197Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0359777Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0361068Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0362290Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0363491Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0364696Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0365904Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0367115Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0368461Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0369739Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0370966Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0372176Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0373388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0374603Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0375839Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0377034Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0378236Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0379444Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0380708Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0381904Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0383120Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0384452Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0385678Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0386887Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0388105Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0389316Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0390522Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0391744Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0392935Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0394140Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0394866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0395353Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0396341Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0397558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0398826Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0400077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0400589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0401583Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0402814Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0404018Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0405243Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0406457Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0407671Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0408877Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0410088Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0410817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0411286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0411760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0412235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0413219Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0414534Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0415789Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0417025Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0417749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0418231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0419220Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0420434Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0421643Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0422860Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0424069Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0425278Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0426495Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0427707Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0428987Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0430246Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0431450Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0432671Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0433384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0433870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0434850Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0436061Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0437320Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0438539Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0440047Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0441285Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0442500Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0443218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0443803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0444775Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0446059Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0446798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0447272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0448265Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0449470Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0450696Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0451922Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0453147Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0453869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0454348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0455312Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0456540Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0457261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0457741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0458726Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0460008Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0461270Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0462506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0463731Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0464945Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0465662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0466142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0467119Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0468337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0469538Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0470765Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0471976Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0473180Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0474454Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0475698Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0476927Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0478154Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0479538Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0480803Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0482021Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0483244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0484454Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0485175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0485638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0486625Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0487840Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0489057Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0489868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0490334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:13:18.0491386Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0491991Z dist init r=0, world=2
2023-01-11T22:13:18.0492241Z dist init r=1, world=2
2023-01-11T22:13:18.0492461Z ok (23.079s)
2023-01-11T22:13:18.0492763Z   test_low_precision_grads (__main__.TestClipGradNorm)
2023-01-11T22:13:18.0493260Z Tests ``clip_grad_norm_()`` when using low precision gradients. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51503
2023-01-11T22:13:18.0493767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51504
2023-01-11T22:13:18.0494379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:18.0494830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:18.0495406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:18.0495855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:18.0496430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:18.0496871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:18.0497429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:18.0497890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:18.0498339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:18.0498827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:18.0499467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:18.0500152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:18.0500671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:18.0501136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:18.0502119Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0503362Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0504589Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0505887Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0507160Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0508395Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0509611Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0510831Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0512034Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0513249Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0514462Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0515677Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0516897Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0518117Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0519325Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0520860Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0522133Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0523356Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0524575Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0525788Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0526987Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0528208Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0529428Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0530636Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0531849Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0533053Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0534270Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0535568Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0536824Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0538047Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0539258Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0540471Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0541679Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0542882Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0544101Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0545311Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0546520Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0547731Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0548936Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0550214Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0551479Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0552702Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0553899Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0555120Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0556329Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:13:18.0556925Z dist init r=0, world=2
2023-01-11T22:13:18.0557176Z dist init r=1, world=2
2023-01-11T22:13:18.0557395Z ok (4.912s)
2023-01-11T22:13:18.0557671Z   test_non_root (__main__.TestClipGradNorm)
2023-01-11T22:13:18.0558291Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51586
2023-01-11T22:13:18.0558810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51587
2023-01-11T22:13:18.0559593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:18.0560056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:18.0560633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:18.0561081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:18.0561664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:18.0562105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:18.0562679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:18.0563123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:18.0563573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:18.0564068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:18.0564707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:18.0565389Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:18.0566001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:18.0566470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:18.0566808Z dist init r=1, world=2
2023-01-11T22:13:18.0567058Z dist init r=0, world=2
2023-01-11T22:13:18.0567295Z ok (4.512s)
2023-01-11T22:13:18.0567446Z 
2023-01-11T22:13:18.0567703Z ----------------------------------------------------------------------
2023-01-11T22:13:18.0568030Z Ran 3 tests in 32.504s
2023-01-11T22:13:18.0568259Z 
2023-01-11T22:13:18.0568360Z OK
2023-01-11T22:13:18.0568496Z 
2023-01-11T22:13:18.0568620Z Generating XML reports...
2023-01-11T22:13:18.0569216Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20230111221245.xml
2023-01-11T22:13:18.0569574Z 
2023-01-11T22:13:18.0569922Z ##[endgroup]
2023-01-11T22:13:18.0570549Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_mpnp2inh)
2023-01-11T22:13:18.0570907Z 
2023-01-11T22:13:18.0571223Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2023-01-11 22:13:18.013567]
2023-01-11T22:13:18.0571956Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:13:18.013820]
2023-01-11T22:13:54.3138486Z 
2023-01-11T22:13:54.3139374Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T22:13:54.3140385Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_e5wpdbfx)
2023-01-11T22:13:54.3140804Z 
2023-01-11T22:13:54.3140898Z Running tests...
2023-01-11T22:13:54.3143819Z ----------------------------------------------------------------------
2023-01-11T22:13:54.3144479Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2023-01-11T22:13:54.3145152Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:13:54.3145900Z This unit test verifies the ``allreduce`` hook registered case gives same result ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:13:54.3146428Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51704
2023-01-11T22:13:54.3146854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51705
2023-01-11T22:13:54.3147491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3147956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3148500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3150877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3151594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3152055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3152637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3153121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3153556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:54.3154055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:54.3154544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:54.3155013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:54.3155938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3156619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3157202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpztcvs_1s
2023-01-11T22:13:54.3157827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpztcvs_1s/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3158389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplksjwigo
2023-01-11T22:13:54.3158918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplksjwigo/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3159292Z ok (6.556s)
2023-01-11T22:13:54.3159991Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:13:54.3160576Z This unit test verifies the ``fp16 compress`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51788
2023-01-11T22:13:54.3161119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51789
2023-01-11T22:13:54.3161751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3162201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3162790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3163263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3163831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3164277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3164838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3165304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3165728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:54.3166197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:54.3166688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:54.3167235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:54.3167894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3168568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3169124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8smfsfg0
2023-01-11T22:13:54.3169663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8smfsfg0/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3170177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6qs110y
2023-01-11T22:13:54.3170708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6qs110y/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3171091Z ok (5.012s)
2023-01-11T22:13:54.3171456Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:13:54.3172028Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51872
2023-01-11T22:13:54.3172578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51873
2023-01-11T22:13:54.3173306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3173759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3174325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3174794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3175438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3175881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3176463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3176929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3177368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:54.3177828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:54.3178314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:54.3178807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:54.3179451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3180144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3180698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2h4rq50g
2023-01-11T22:13:54.3181234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2h4rq50g/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3181746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ijl3u66
2023-01-11T22:13:54.3182277Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ijl3u66/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3182656Z ok (5.113s)
2023-01-11T22:13:54.3183045Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:13:54.3183668Z This unit test verifies the ``quantize per channel`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51956
2023-01-11T22:13:54.3184212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51957
2023-01-11T22:13:54.3184825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3185254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3185836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3186308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3186890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3187317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3187888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3202046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3202521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:54.3202981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:54.3203468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:54.3204131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:54.3204849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3205533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3206087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpooke_n0d
2023-01-11T22:13:54.3206709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpooke_n0d/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3207244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq_u5nbv
2023-01-11T22:13:54.3207780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq_u5nbv/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3208164Z ok (5.014s)
2023-01-11T22:13:54.3208545Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:13:54.3209123Z This unit test verifies the ``quantize per tensor`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52040
2023-01-11T22:13:54.3209665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52041
2023-01-11T22:13:54.3210288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3210744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3211299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3211747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3212323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3212780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3213375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3213838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3214273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:54.3214729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:54.3215218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:54.3215718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:54.3216359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3217053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3217607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpodyt5r8p
2023-01-11T22:13:54.3218129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpodyt5r8p/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3218649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0_ri1zl
2023-01-11T22:13:54.3219185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0_ri1zl/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3219568Z ok (5.012s)
2023-01-11T22:13:54.3220023Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52124
2023-01-11T22:13:54.3220581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52125
2023-01-11T22:13:54.3221192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3221713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3222264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:13:54.3222714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:13:54.3223288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3223811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3224391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:13:54.3224858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:13:54.3225296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:13:54.3225749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:13:54.3226235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:13:54.3226732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:13:54.3227390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3228064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:13:54.3228612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqim9ewm
2023-01-11T22:13:54.3229152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqim9ewm/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3229683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzio5y5cj
2023-01-11T22:13:54.3230207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzio5y5cj/_remote_module_non_scriptable.py
2023-01-11T22:13:54.3230595Z ok (7.317s)
2023-01-11T22:13:54.3230749Z 
2023-01-11T22:13:54.3231020Z ----------------------------------------------------------------------
2023-01-11T22:13:54.3231335Z Ran 6 tests in 34.025s
2023-01-11T22:13:54.3231501Z 
2023-01-11T22:13:54.3231599Z OK
2023-01-11T22:13:54.3231734Z 
2023-01-11T22:13:54.3231858Z Generating XML reports...
2023-01-11T22:13:54.3232579Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20230111221319.xml
2023-01-11T22:13:54.3233006Z 
2023-01-11T22:13:54.3233468Z ##[endgroup]
2023-01-11T22:13:54.3234158Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_e5wpdbfx)
2023-01-11T22:13:54.3234574Z 
2023-01-11T22:13:54.3234853Z Running distributed/_composable/test_compose ... [2023-01-11 22:13:54.314080]
2023-01-11T22:13:54.3235549Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_compose.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:13:54.314440]
2023-01-11T22:14:34.1200184Z 
2023-01-11T22:14:34.1200919Z Expand the folded group to see the log file of distributed/_composable/test_compose
2023-01-11T22:14:34.1201881Z ##[group]PRINTING LOG FILE of distributed/_composable/test_compose (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_compose_j6vdcp4b)
2023-01-11T22:14:34.1205388Z 
2023-01-11T22:14:34.1205763Z Running tests...
2023-01-11T22:14:34.1206322Z ----------------------------------------------------------------------
2023-01-11T22:14:34.1206906Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_compose
2023-01-11T22:14:34.1207460Z   test_checkpoint_fsdp_submodules_non_reentrant (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:14:34.1208274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52243
2023-01-11T22:14:34.1208740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52244
2023-01-11T22:14:34.1209410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1212680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1213452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1213966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1214566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1214999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1215577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1216042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1216516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1216985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1217652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1218352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1218864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1219320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1219682Z dist init r=1, world=2
2023-01-11T22:14:34.1219954Z dist init r=0, world=2
2023-01-11T22:14:34.1220179Z ok (6.213s)
2023-01-11T22:14:34.1220659Z   test_checkpoint_fsdp_submodules_use_reentrant (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52326
2023-01-11T22:14:34.1221216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52327
2023-01-11T22:14:34.1221837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1222271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1222840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1223314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1223906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1224348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1224901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1225368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1225810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1226324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1226982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1227674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1228248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1228711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1229067Z dist init r=1, world=2
2023-01-11T22:14:34.1229302Z dist init r=0, world=2
2023-01-11T22:14:34.1229544Z ok (4.411s)
2023-01-11T22:14:34.1229999Z   test_checkpoint_fsdp_submodules_with_param (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52409
2023-01-11T22:14:34.1230593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52410
2023-01-11T22:14:34.1231196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1231645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1232216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1232686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1233246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1233687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1234335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1234798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1235251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1235728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1236381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1237069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1237592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1238042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1238389Z dist init r=0, world=2
2023-01-11T22:14:34.1238641Z dist init r=1, world=2
2023-01-11T22:14:34.1238863Z ok (4.511s)
2023-01-11T22:14:34.1239333Z   test_checkpoint_fsdp_submodules_with_param_no_shard (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52492
2023-01-11T22:14:34.1240263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52493
2023-01-11T22:14:34.1240880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1241308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1241882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1242347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1242922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1243356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1243932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1244394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1244825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1245315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1246071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1246761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1247257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1247726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1248145Z dist init r=0, world=2
2023-01-11T22:14:34.1248389Z dist init r=1, world=2
2023-01-11T22:14:34.1248627Z ok (4.512s)
2023-01-11T22:14:34.1249071Z   test_composable_fsdp_replicate (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52575
2023-01-11T22:14:34.1249588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52576
2023-01-11T22:14:34.1250207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1250654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1251225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1251686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1252254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1252696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1253264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1253707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1254170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1254655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1255289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1255970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1256493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1256964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1258216Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:14:34.1259072Z   warnings.warn(
2023-01-11T22:14:34.1260214Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:14:34.1260988Z   warnings.warn(
2023-01-11T22:14:34.1261236Z dist init r=0, world=2
2023-01-11T22:14:34.1261486Z dist init r=1, world=2
2023-01-11T22:14:34.1261708Z ok (3.910s)
2023-01-11T22:14:34.1262058Z   test_fully_shard_replicate_composability (__main__.TestFSDPCheckpoint)
2023-01-11T22:14:34.1262645Z Tests composing ``fully_shard`` and ``replicate``. To save unit test ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52654
2023-01-11T22:14:34.1263153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52655
2023-01-11T22:14:34.1263766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1264219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1264842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1265298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1265875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1266316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1266873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1267337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1267791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1268286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1268921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1269603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1270118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1270586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1271045Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1271519Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1271987Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1272430Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1272902Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1273360Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1273816Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1274253Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1274722Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1275184Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1275624Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1276082Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1276546Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1277008Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1277444Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1277906Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1278366Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1278892Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1279327Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1280061Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1280519Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1281035Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1281512Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1281975Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1282434Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1282875Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:14:34.1283224Z dist init r=0, world=2
2023-01-11T22:14:34.1283474Z dist init r=1, world=2
2023-01-11T22:14:34.1283697Z ok (4.912s)
2023-01-11T22:14:34.1284155Z   test_wrap_same_submodule_use_reentrant_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52737
2023-01-11T22:14:34.1284703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52738
2023-01-11T22:14:34.1285329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1285759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1286331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1286838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1287408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1287853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1288417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1288877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1289313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1289809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1290467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1291133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1291657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1292127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1292482Z dist init r=0, world=2
2023-01-11T22:14:34.1292715Z dist init r=1, world=2
2023-01-11T22:14:34.1292954Z ok (4.512s)
2023-01-11T22:14:34.1293416Z   test_wrap_same_submodule_use_reentrant_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52820
2023-01-11T22:14:34.1293941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52821
2023-01-11T22:14:34.1294550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1294994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1295652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1296100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1296672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:14:34.1297111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:14:34.1297725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:14:34.1298195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:14:34.1298646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:14:34.1299138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:14:34.1299780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1300467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:14:34.1300983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:14:34.1301451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:14:34.1301784Z dist init r=0, world=2
2023-01-11T22:14:34.1302037Z dist init r=1, world=2
2023-01-11T22:14:34.1302277Z ok (4.512s)
2023-01-11T22:14:34.1302424Z 
2023-01-11T22:14:34.1302675Z ----------------------------------------------------------------------
2023-01-11T22:14:34.1303002Z Ran 8 tests in 37.494s
2023-01-11T22:14:34.1303161Z 
2023-01-11T22:14:34.1303256Z OK
2023-01-11T22:14:34.1303389Z 
2023-01-11T22:14:34.1303496Z Generating XML reports...
2023-01-11T22:14:34.1304103Z Generated XML report: test-reports/python-unittest/distributed._composable.test_compose/TEST-TestFSDPCheckpoint-20230111221356.xml
2023-01-11T22:14:34.1304461Z 
2023-01-11T22:14:34.1304856Z ##[endgroup]
2023-01-11T22:14:34.1305470Z FINISHED PRINTING LOG FILE of distributed/_composable/test_compose (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_compose_j6vdcp4b)
2023-01-11T22:14:34.1305808Z 
2023-01-11T22:14:34.1306123Z Running distributed/checkpoint/test_file_system_checkpoint_cpu ... [2023-01-11 22:14:34.120152]
2023-01-11T22:14:34.1306872Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:14:34.120402]
2023-01-11T22:15:15.4654127Z 
2023-01-11T22:15:15.4654700Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T22:15:15.4658883Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_vdmj65ju)
2023-01-11T22:15:15.4659355Z 
2023-01-11T22:15:15.4659456Z Running tests...
2023-01-11T22:15:15.4660188Z ----------------------------------------------------------------------
2023-01-11T22:15:15.4660808Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu
2023-01-11T22:15:15.4661383Z   test_load_rowwise_to_colwise_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:15:15.4661891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52938
2023-01-11T22:15:15.4662350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52939
2023-01-11T22:15:15.4662966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4663394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4664281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4664735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4665319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4665751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4666403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4666881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4667321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4670246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4670744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4671295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4672004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4672709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4673866Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4674676Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4675734Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4676478Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4676758Z ok (5.539s)
2023-01-11T22:15:15.4677215Z   test_load_rowwise_to_colwise_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53014
2023-01-11T22:15:15.4677787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53015
2023-01-11T22:15:15.4678392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4678844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4679755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4680248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4680836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4681284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4681854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4682310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4682743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4683198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4683874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4684374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4685039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4685707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4686965Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4687729Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4688790Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4689522Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4689775Z ok (4.011s)
2023-01-11T22:15:15.4690256Z   test_load_with_different_shard_plan_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53092
2023-01-11T22:15:15.4690879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53093
2023-01-11T22:15:15.4691565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4692006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4692581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4693027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4693602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4694042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4694596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4695056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4695484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4695958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4696420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4696911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4697563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4698244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4699396Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4700134Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4701256Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4702002Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4702246Z ok (4.112s)
2023-01-11T22:15:15.4702786Z   test_load_with_different_shard_plan_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53168
2023-01-11T22:15:15.4703376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53169
2023-01-11T22:15:15.4703977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4704410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4704981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4705445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4706022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4706443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4707016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4707472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4707886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4708352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4708832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4709331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4709960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4710642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4711812Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4712551Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4713597Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4714311Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4714584Z ok (4.113s)
2023-01-11T22:15:15.4715051Z   test_save_load_bytes_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53268
2023-01-11T22:15:15.4715584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53269
2023-01-11T22:15:15.4716191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4716709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4717283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4717733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4718301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4718736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4719360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4720139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4720573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4721043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4721515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4722019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4722669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4723354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4723730Z ok (2.408s)
2023-01-11T22:15:15.4724192Z   test_save_load_bytes_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53342
2023-01-11T22:15:15.4724741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53343
2023-01-11T22:15:15.4725346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4725777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4726348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4726811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4727367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4727806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4728374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4728829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4729248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4729718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4730202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4730670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4731323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4732003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4732387Z ok (2.408s)
2023-01-11T22:15:15.4732862Z   test_switch_between_sharded_tensor_to_tensor_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53418
2023-01-11T22:15:15.4733436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53419
2023-01-11T22:15:15.4734145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4734588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4735141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4735604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4736240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4736671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4737242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4737704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4738138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4738588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4739065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4739551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4740209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4740878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4742044Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4742784Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4743829Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4744558Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4744811Z ok (4.212s)
2023-01-11T22:15:15.4745304Z   test_switch_between_sharded_tensor_to_tensor_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53494
2023-01-11T22:15:15.4745884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53495
2023-01-11T22:15:15.4746479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4746922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4747494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4747958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4748520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4748957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4749525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4749984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4750470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4750939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4751420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4751891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4752600Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4753295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4754462Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4755199Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4756225Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4756948Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4757224Z ok (4.212s)
2023-01-11T22:15:15.4758420Z   test_read_write_only_tensor_thread_count_1 (__main__.TestDistributedStateDictSaveLoad) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4759262Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4759841Z ok (0.044s)
2023-01-11T22:15:15.4760224Z   test_read_write_only_tensor_thread_count_2 (__main__.TestDistributedStateDictSaveLoad) ... ok (0.016s)
2023-01-11T22:15:15.4760904Z   test_read_write_shard_tensor_thread_count_1 (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53603
2023-01-11T22:15:15.4761509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53604
2023-01-11T22:15:15.4762122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4762566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4763140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4763588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4764163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4764601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4765153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4765609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4766040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4766508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4767072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4767564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4768221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4768898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4770140Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4770885Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4771947Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4772677Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4772932Z ok (3.910s)
2023-01-11T22:15:15.4773471Z   test_read_write_shard_tensor_thread_count_2 (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53679
2023-01-11T22:15:15.4774100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53680
2023-01-11T22:15:15.4774705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4775134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4775703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4776162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4776735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:15:15.4777154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:15:15.4777721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:15:15.4778175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:15:15.4778588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:15:15.4779051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:15:15.4779528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:15:15.4780019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:15:15.4780653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4781331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:15:15.4782490Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4783297Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4784353Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:15:15.4785063Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:15:15.4785336Z ok (4.011s)
2023-01-11T22:15:15.4785535Z 
2023-01-11T22:15:15.4785813Z ----------------------------------------------------------------------
2023-01-11T22:15:15.4786125Z Ran 12 tests in 38.999s
2023-01-11T22:15:15.4786288Z 
2023-01-11T22:15:15.4786384Z OK
2023-01-11T22:15:15.4786514Z 
2023-01-11T22:15:15.4786637Z Generating XML reports...
2023-01-11T22:15:15.4787322Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20230111221436.xml
2023-01-11T22:15:15.4788248Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20230111221436.xml
2023-01-11T22:15:15.4789266Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111221436.xml
2023-01-11T22:15:15.4789752Z 
2023-01-11T22:15:15.4790137Z ##[endgroup]
2023-01-11T22:15:15.4790883Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_vdmj65ju)
2023-01-11T22:15:15.4791281Z 
2023-01-11T22:15:15.4791545Z Running distributed/test_c10d_spawn_gloo ... [2023-01-11 22:15:15.465764]
2023-01-11T22:15:15.4792247Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:15:15.466084]
2023-01-11T22:16:46.8001185Z 
2023-01-11T22:16:46.8001852Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo
2023-01-11T22:16:46.8002751Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_fsrmtvz6)
2023-01-11T22:16:46.8007155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmhs2w59a
2023-01-11T22:16:46.8007760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmhs2w59a/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8008204Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8009494Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2023-01-11T22:16:46.8010357Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T22:16:46.8010798Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T22:16:46.8011268Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T22:16:46.8011650Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:16:46.8011975Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:16:46.8013524Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2023-01-11T22:16:46.8014940Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8015322Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8015749Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8016154Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8016554Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8016907Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8017302Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8017772Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:16:46.8018499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8018940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8019523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8019999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8020481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf49txu4s
2023-01-11T22:16:46.8021031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf49txu4s/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8021468Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8021660Z 
2023-01-11T22:16:46.8021759Z Running tests...
2023-01-11T22:16:46.8022150Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8022716Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8023333Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8024070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:16:46.8024605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:16:46.8025063Z ok (0.017s)
2023-01-11T22:16:46.8025349Z 
2023-01-11T22:16:46.8025832Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8026360Z Ran 1 test in 0.018s
2023-01-11T22:16:46.8026662Z 
2023-01-11T22:16:46.8026800Z OK
2023-01-11T22:16:46.8027011Z 
2023-01-11T22:16:46.8027308Z Generating XML reports...
2023-01-11T22:16:46.8028593Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111221522.xml
2023-01-11T22:16:46.8030016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8030788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8031955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8032882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8033313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujdtn3yb
2023-01-11T22:16:46.8033853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujdtn3yb/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8034280Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8034478Z 
2023-01-11T22:16:46.8034569Z Running tests...
2023-01-11T22:16:46.8034984Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8035525Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8036148Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8037033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:16:46.8037558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:16:46.8037911Z ok (0.458s)
2023-01-11T22:16:46.8038060Z 
2023-01-11T22:16:46.8038329Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8038634Z Ran 1 test in 0.458s
2023-01-11T22:16:46.8038793Z 
2023-01-11T22:16:46.8038887Z OK
2023-01-11T22:16:46.8039022Z 
2023-01-11T22:16:46.8039207Z Generating XML reports...
2023-01-11T22:16:46.8040454Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111221526.xml
2023-01-11T22:16:46.8041237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8041690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8042269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8042721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8043187Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpf2tjap
2023-01-11T22:16:46.8043728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpf2tjap/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8044134Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8044333Z 
2023-01-11T22:16:46.8044440Z Running tests...
2023-01-11T22:16:46.8044846Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8045383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8045984Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8046760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:16:46.8047281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:16:46.8047612Z ok (1.245s)
2023-01-11T22:16:46.8047758Z 
2023-01-11T22:16:46.8048017Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8048357Z Ran 1 test in 1.246s
2023-01-11T22:16:46.8048499Z 
2023-01-11T22:16:46.8048597Z OK
2023-01-11T22:16:46.8048712Z 
2023-01-11T22:16:46.8048836Z Generating XML reports...
2023-01-11T22:16:46.8049522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111221530.xml
2023-01-11T22:16:46.8050294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8050761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8051311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8051774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8052237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5r89ilp2
2023-01-11T22:16:46.8052758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5r89ilp2/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8053186Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8053384Z 
2023-01-11T22:16:46.8053494Z Running tests...
2023-01-11T22:16:46.8053875Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8054412Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8054984Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53943
2023-01-11T22:16:46.8055645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53944
2023-01-11T22:16:46.8056240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8056689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8057246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8057781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8058336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8058798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8059385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8059831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8060304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp8bhrv4
2023-01-11T22:16:46.8060831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp8bhrv4/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8061363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mqli9f9
2023-01-11T22:16:46.8061879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mqli9f9/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8062305Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8062627Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8063012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8063484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8063968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8064463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8065102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8065783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8066179Z ok (5.327s)
2023-01-11T22:16:46.8066325Z 
2023-01-11T22:16:46.8066592Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8066896Z Ran 1 test in 5.327s
2023-01-11T22:16:46.8067056Z 
2023-01-11T22:16:46.8067149Z OK
2023-01-11T22:16:46.8067279Z 
2023-01-11T22:16:46.8067402Z Generating XML reports...
2023-01-11T22:16:46.8068021Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221536.xml
2023-01-11T22:16:46.8068768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8069218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8069794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8070241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8070709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lwi3s8e
2023-01-11T22:16:46.8071245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lwi3s8e/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8071651Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8071849Z 
2023-01-11T22:16:46.8071957Z Running tests...
2023-01-11T22:16:46.8072357Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8072965Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8073514Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54059
2023-01-11T22:16:46.8074048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54060
2023-01-11T22:16:46.8074701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8075137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8075712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8076306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8076885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8077309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8077878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8078336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8078798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2uq25o3
2023-01-11T22:16:46.8079316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2uq25o3/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8080297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdrs2ogy
2023-01-11T22:16:46.8080834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdrs2ogy/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8081240Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8081644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8082042Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8082440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8082909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8083397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8084069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8084741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8085135Z ok (5.327s)
2023-01-11T22:16:46.8085282Z 
2023-01-11T22:16:46.8085544Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8085876Z Ran 1 test in 5.327s
2023-01-11T22:16:46.8086021Z 
2023-01-11T22:16:46.8086112Z OK
2023-01-11T22:16:46.8086245Z 
2023-01-11T22:16:46.8086369Z Generating XML reports...
2023-01-11T22:16:46.8087013Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221545.xml
2023-01-11T22:16:46.8087730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8088178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8088753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8089219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8089665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpve3dffge
2023-01-11T22:16:46.8090204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpve3dffge/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8090736Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8090932Z 
2023-01-11T22:16:46.8091023Z Running tests...
2023-01-11T22:16:46.8091427Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8091963Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8092537Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54175
2023-01-11T22:16:46.8093128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54176
2023-01-11T22:16:46.8093745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8094191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8094766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8095218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8095790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8096231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8096784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8097250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8097709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2lmcwcd
2023-01-11T22:16:46.8098301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2lmcwcd/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8098816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9rqm98j
2023-01-11T22:16:46.8099352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9rqm98j/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8099778Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8100084Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8100485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8100953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8101445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8101918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8102576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8103258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8103655Z ok (5.327s)
2023-01-11T22:16:46.8103787Z 
2023-01-11T22:16:46.8104053Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8104381Z Ran 1 test in 5.327s
2023-01-11T22:16:46.8104541Z 
2023-01-11T22:16:46.8104636Z OK
2023-01-11T22:16:46.8104751Z 
2023-01-11T22:16:46.8104879Z Generating XML reports...
2023-01-11T22:16:46.8105520Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221554.xml
2023-01-11T22:16:46.8106258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8106705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8107261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8107720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8108251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw4r8kli
2023-01-11T22:16:46.8108773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw4r8kli/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8109195Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8109393Z 
2023-01-11T22:16:46.8109504Z Running tests...
2023-01-11T22:16:46.8109891Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8110493Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8111076Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54291
2023-01-11T22:16:46.8111618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54292
2023-01-11T22:16:46.8112209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8112659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8113234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8113697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8114254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8114698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8115265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8115706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8116169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptjmew9t
2023-01-11T22:16:46.8116711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptjmew9t/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8117239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpducr91qh
2023-01-11T22:16:46.8117749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpducr91qh/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8118175Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8118494Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8118882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8119351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8120291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8120786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8121442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8122124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8122517Z ok (5.327s)
2023-01-11T22:16:46.8122660Z 
2023-01-11T22:16:46.8122931Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8123236Z Ran 1 test in 5.327s
2023-01-11T22:16:46.8123394Z 
2023-01-11T22:16:46.8123486Z OK
2023-01-11T22:16:46.8123619Z 
2023-01-11T22:16:46.8123749Z Generating XML reports...
2023-01-11T22:16:46.8124372Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221603.xml
2023-01-11T22:16:46.8125110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8125551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8126231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8126683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8127143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09x3_m02
2023-01-11T22:16:46.8127672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09x3_m02/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8128137Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8128344Z 
2023-01-11T22:16:46.8128453Z Running tests...
2023-01-11T22:16:46.8128861Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8129397Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8129945Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54407
2023-01-11T22:16:46.8130489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54408
2023-01-11T22:16:46.8131096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8131526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8132099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8132569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8133154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8133573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8134140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8134608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8135069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhkg1man
2023-01-11T22:16:46.8135592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhkg1man/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8136121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4b0amxx
2023-01-11T22:16:46.8136655Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4b0amxx/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8137060Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8137381Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8137785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8138237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8138724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8139210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8139862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8140526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8140916Z ok (5.326s)
2023-01-11T22:16:46.8141060Z 
2023-01-11T22:16:46.8141330Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8141654Z Ran 1 test in 5.326s
2023-01-11T22:16:46.8141795Z 
2023-01-11T22:16:46.8141889Z OK
2023-01-11T22:16:46.8142021Z 
2023-01-11T22:16:46.8142147Z Generating XML reports...
2023-01-11T22:16:46.8142783Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221613.xml
2023-01-11T22:16:46.8143588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8144037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8144609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8145078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8145578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ef92sv0
2023-01-11T22:16:46.8146117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ef92sv0/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8146539Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8146733Z 
2023-01-11T22:16:46.8146825Z Running tests...
2023-01-11T22:16:46.8147230Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8147770Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8148330Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54523
2023-01-11T22:16:46.8148844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54524
2023-01-11T22:16:46.8149443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8149891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8150445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8150919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8151490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8151932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8152484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8152946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8153406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ziczrbu
2023-01-11T22:16:46.8153945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ziczrbu/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8154456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiedpizaj
2023-01-11T22:16:46.8154987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiedpizaj/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8155415Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8155717Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8156123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8156587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8157073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8157546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8158204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8158896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8159268Z ok (5.327s)
2023-01-11T22:16:46.8159761Z 
2023-01-11T22:16:46.8160114Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8160446Z Ran 1 test in 5.328s
2023-01-11T22:16:46.8160705Z 
2023-01-11T22:16:46.8160799Z OK
2023-01-11T22:16:46.8160913Z 
2023-01-11T22:16:46.8161038Z Generating XML reports...
2023-01-11T22:16:46.8161681Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221622.xml
2023-01-11T22:16:46.8162417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8162848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8163483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8163959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8164424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp2c4z0g
2023-01-11T22:16:46.8164947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp2c4z0g/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8165379Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8165574Z 
2023-01-11T22:16:46.8165683Z Running tests...
2023-01-11T22:16:46.8166069Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8166616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8167184Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54639
2023-01-11T22:16:46.8167723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54640
2023-01-11T22:16:46.8168314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8168764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8169338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8169807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8170364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8170807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8171376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8171816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8172280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1l1wi7jn
2023-01-11T22:16:46.8172814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1l1wi7jn/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8173342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3q2c7faq
2023-01-11T22:16:46.8173849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3q2c7faq/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8174280Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8174681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8175054Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8175453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8175936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8176426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8177063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8177751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8178213Z ok (5.326s)
2023-01-11T22:16:46.8178360Z 
2023-01-11T22:16:46.8178610Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8178936Z Ran 1 test in 5.327s
2023-01-11T22:16:46.8179097Z 
2023-01-11T22:16:46.8179192Z OK
2023-01-11T22:16:46.8179325Z 
2023-01-11T22:16:46.8179448Z Generating XML reports...
2023-01-11T22:16:46.8180069Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221631.xml
2023-01-11T22:16:46.8180859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8181316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8181875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8182339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8182808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1glshax
2023-01-11T22:16:46.8183352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1glshax/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8183759Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8183952Z 
2023-01-11T22:16:46.8184063Z Running tests...
2023-01-11T22:16:46.8184464Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8184982Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:16:46.8185551Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54755
2023-01-11T22:16:46.8186084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54756
2023-01-11T22:16:46.8186682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8187117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8187690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8188154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8188726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:16:46.8189147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:16:46.8189710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:16:46.8190167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:16:46.8190611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswadzk8k
2023-01-11T22:16:46.8191153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswadzk8k/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8191687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzernjk5r
2023-01-11T22:16:46.8192217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzernjk5r/_remote_module_non_scriptable.py
2023-01-11T22:16:46.8192621Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8193023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:16:46.8193410Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:16:46.8193798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:16:46.8194286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:16:46.8194779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:16:46.8195440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8196178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:16:46.8196566Z ok (5.328s)
2023-01-11T22:16:46.8196712Z 
2023-01-11T22:16:46.8196977Z ----------------------------------------------------------------------
2023-01-11T22:16:46.8197283Z Ran 1 test in 5.328s
2023-01-11T22:16:46.8197449Z 
2023-01-11T22:16:46.8197543Z OK
2023-01-11T22:16:46.8197675Z 
2023-01-11T22:16:46.8197800Z Generating XML reports...
2023-01-11T22:16:46.8198550Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221640.xml
2023-01-11T22:16:46.8198943Z 
2023-01-11T22:16:46.8199318Z ##[endgroup]
2023-01-11T22:16:46.8200374Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_fsrmtvz6)
2023-01-11T22:16:46.8200722Z 
2023-01-11T22:16:46.8200990Z Running distributed/test_c10d_spawn_nccl ... [2023-01-11 22:16:46.800694]
2023-01-11T22:16:46.8201672Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:16:46.801040]
2023-01-11T22:18:14.4527357Z 
2023-01-11T22:18:14.4528428Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2023-01-11T22:18:14.4530181Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_ooxwhz4g)
2023-01-11T22:18:14.4531199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfohf_m6n
2023-01-11T22:18:14.4533524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfohf_m6n/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4534020Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4534347Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:18:14.4534683Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:18:14.4535998Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2023-01-11T22:18:14.4537316Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4537723Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4538130Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4538561Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4538966Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4539349Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4539741Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4540145Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4540564Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:18:14.4541323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4541785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4542366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4542824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4543569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglho23ri
2023-01-11T22:18:14.4544143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglho23ri/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4544577Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4544782Z 
2023-01-11T22:18:14.4544900Z Running tests...
2023-01-11T22:18:14.4545313Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4545973Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4546541Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54905
2023-01-11T22:18:14.4547082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54906
2023-01-11T22:18:14.4547702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4548155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4548711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4549184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4549766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4550218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4550778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4551229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4551697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmtrggb83
2023-01-11T22:18:14.4552249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmtrggb83/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4552764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ruvt_ew
2023-01-11T22:18:14.4553298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ruvt_ew/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4553731Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4554120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4554612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4555018Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4555423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4555893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4556562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4557429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4557820Z ok (5.427s)
2023-01-11T22:18:14.4557952Z 
2023-01-11T22:18:14.4558224Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4558552Z Ran 1 test in 5.427s
2023-01-11T22:18:14.4558712Z 
2023-01-11T22:18:14.4558806Z OK
2023-01-11T22:18:14.4558944Z 
2023-01-11T22:18:14.4559054Z Generating XML reports...
2023-01-11T22:18:14.4560450Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221653.xml
2023-01-11T22:18:14.4561506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4561962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4562663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4563130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4563593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpox3s359u
2023-01-11T22:18:14.4564111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpox3s359u/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4564607Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4564814Z 
2023-01-11T22:18:14.4564923Z Running tests...
2023-01-11T22:18:14.4565330Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4565850Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4566427Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55024
2023-01-11T22:18:14.4566979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55025
2023-01-11T22:18:14.4567571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4568021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4568597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4569069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4569628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4570076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4570646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4571110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4571557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbc6f1ub
2023-01-11T22:18:14.4572100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbc6f1ub/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4572628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv007bae2
2023-01-11T22:18:14.4573143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv007bae2/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4573567Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4573970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4574462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4574844Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4575249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4575731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4576372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4577291Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:18:14.4577839Z   warnings.warn(
2023-01-11T22:18:14.4578374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4579286Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:18:14.4579879Z   warnings.warn(
2023-01-11T22:18:14.4580657Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:3001: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2023-01-11T22:18:14.4581207Z   warnings.warn(
2023-01-11T22:18:14.4582015Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:3001: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2023-01-11T22:18:14.4582545Z   warnings.warn(
2023-01-11T22:18:14.4582781Z ok (5.327s)
2023-01-11T22:18:14.4582929Z 
2023-01-11T22:18:14.4583202Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4583512Z Ran 1 test in 5.327s
2023-01-11T22:18:14.4583676Z 
2023-01-11T22:18:14.4583770Z OK
2023-01-11T22:18:14.4583905Z 
2023-01-11T22:18:14.4584029Z Generating XML reports...
2023-01-11T22:18:14.4584655Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221703.xml
2023-01-11T22:18:14.4585396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4585843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4586422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4586874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4587341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg92af6a3
2023-01-11T22:18:14.4587878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg92af6a3/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4588305Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4588484Z 
2023-01-11T22:18:14.4588592Z Running tests...
2023-01-11T22:18:14.4588996Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4589537Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4590085Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55143
2023-01-11T22:18:14.4590625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55144
2023-01-11T22:18:14.4591228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4591678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4592231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4592703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4593280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4593702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4594272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4594739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4595195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kyaw3ul
2023-01-11T22:18:14.4595723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kyaw3ul/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4596246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuh5_4ukl
2023-01-11T22:18:14.4596772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuh5_4ukl/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4597257Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4597560Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4597963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4598453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4598920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4599811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4600974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4601677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4602059Z ok (5.427s)
2023-01-11T22:18:14.4602209Z 
2023-01-11T22:18:14.4602477Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4602805Z Ran 1 test in 5.427s
2023-01-11T22:18:14.4602966Z 
2023-01-11T22:18:14.4603041Z OK
2023-01-11T22:18:14.4603175Z 
2023-01-11T22:18:14.4603300Z Generating XML reports...
2023-01-11T22:18:14.4603939Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221712.xml
2023-01-11T22:18:14.4604685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4605119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4605739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4606210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4606664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzzykcn2
2023-01-11T22:18:14.4607208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzzykcn2/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4607638Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4607831Z 
2023-01-11T22:18:14.4607941Z Running tests...
2023-01-11T22:18:14.4608331Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4608877Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4609461Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55264
2023-01-11T22:18:14.4609986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55265
2023-01-11T22:18:14.4610596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4611050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4611627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4612079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4612655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4613100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4613675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4614122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4614589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps4ceueav
2023-01-11T22:18:14.4615114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgz4uesm
2023-01-11T22:18:14.4615734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps4ceueav/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4616281Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgz4uesm/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4616706Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4617110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4617624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4618037Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4618442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4618912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4619578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4620271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4620662Z ok (5.427s)
2023-01-11T22:18:14.4620792Z 
2023-01-11T22:18:14.4621055Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4621379Z Ran 1 test in 5.427s
2023-01-11T22:18:14.4621537Z 
2023-01-11T22:18:14.4621631Z OK
2023-01-11T22:18:14.4621763Z 
2023-01-11T22:18:14.4621874Z Generating XML reports...
2023-01-11T22:18:14.4622519Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221721.xml
2023-01-11T22:18:14.4623263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4623714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4624273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4624738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4625202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_zzw3j2
2023-01-11T22:18:14.4625717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_zzw3j2/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4626145Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4626339Z 
2023-01-11T22:18:14.4626455Z Running tests...
2023-01-11T22:18:14.4626861Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4627381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4627953Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55385
2023-01-11T22:18:14.4628499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55386
2023-01-11T22:18:14.4629108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4629532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4630101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4630568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4631129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4631572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4632139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4632670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4633116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjpd1pqs
2023-01-11T22:18:14.4633660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjpd1pqs/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4634192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ecfab_4
2023-01-11T22:18:14.4634703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ecfab_4/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4635175Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4635590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4636076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4636458Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4636859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4637346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4637989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4638675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4639068Z ok (5.333s)
2023-01-11T22:18:14.4639219Z 
2023-01-11T22:18:14.4639938Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4640524Z Ran 1 test in 5.334s
2023-01-11T22:18:14.4640720Z 
2023-01-11T22:18:14.4640815Z OK
2023-01-11T22:18:14.4640949Z 
2023-01-11T22:18:14.4641075Z Generating XML reports...
2023-01-11T22:18:14.4641711Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221731.xml
2023-01-11T22:18:14.4642461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4642911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4643486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4643935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4644403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pobfcaq
2023-01-11T22:18:14.4644945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pobfcaq/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4645372Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4645549Z 
2023-01-11T22:18:14.4645660Z Running tests...
2023-01-11T22:18:14.4646064Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4646603Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4647162Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55504
2023-01-11T22:18:14.4647703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55505
2023-01-11T22:18:14.4648311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4648762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4649323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4649788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4650361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4650890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4651464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4651923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4652386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrew0ycm
2023-01-11T22:18:14.4652904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrew0ycm/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4653516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgsp_o0od
2023-01-11T22:18:14.4654060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgsp_o0od/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4654468Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4654873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4655365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4655766Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4656150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4656634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4657301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4657992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4658366Z ok (5.428s)
2023-01-11T22:18:14.4658514Z 
2023-01-11T22:18:14.4658778Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4659107Z Ran 1 test in 5.428s
2023-01-11T22:18:14.4659267Z 
2023-01-11T22:18:14.4659343Z OK
2023-01-11T22:18:14.4659482Z 
2023-01-11T22:18:14.4659607Z Generating XML reports...
2023-01-11T22:18:14.4660250Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221740.xml
2023-01-11T22:18:14.4660992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4661425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4662003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4662471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4662916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppblxz7vf
2023-01-11T22:18:14.4663450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppblxz7vf/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4663876Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4664074Z 
2023-01-11T22:18:14.4664184Z Running tests...
2023-01-11T22:18:14.4664568Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4665107Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4665671Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55623
2023-01-11T22:18:14.4666188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55624
2023-01-11T22:18:14.4666798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4667245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4667816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4668331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4668911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4669353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4669903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4670368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4670884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgyjsalp
2023-01-11T22:18:14.4671440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgyjsalp/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4671954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpppyuqyp4
2023-01-11T22:18:14.4672489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpppyuqyp4/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4672921Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4673224Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4673629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4674103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4674591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4675073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4675738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4676422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4676813Z ok (5.427s)
2023-01-11T22:18:14.4676947Z 
2023-01-11T22:18:14.4677213Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4677540Z Ran 1 test in 5.427s
2023-01-11T22:18:14.4677701Z 
2023-01-11T22:18:14.4677795Z OK
2023-01-11T22:18:14.4677929Z 
2023-01-11T22:18:14.4678037Z Generating XML reports...
2023-01-11T22:18:14.4678674Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221749.xml
2023-01-11T22:18:14.4679795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4680645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4681231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4681699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4682167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppr_n3zod
2023-01-11T22:18:14.4682696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppr_n3zod/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4683127Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4683326Z 
2023-01-11T22:18:14.4683437Z Running tests...
2023-01-11T22:18:14.4683840Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4684357Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4684942Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55742
2023-01-11T22:18:14.4685488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55743
2023-01-11T22:18:14.4686074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4686626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4687210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4687678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4688238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4688687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4689320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4689794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4690240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi9656k8
2023-01-11T22:18:14.4690773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi9656k8/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4691302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mjpoxua
2023-01-11T22:18:14.4691818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mjpoxua/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4708792Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4709151Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4709537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4710052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4710544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4711035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4711733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4712429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4712827Z ok (5.428s)
2023-01-11T22:18:14.4712979Z 
2023-01-11T22:18:14.4713249Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4713562Z Ran 1 test in 5.428s
2023-01-11T22:18:14.4713724Z 
2023-01-11T22:18:14.4713818Z OK
2023-01-11T22:18:14.4713950Z 
2023-01-11T22:18:14.4714077Z Generating XML reports...
2023-01-11T22:18:14.4714713Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221758.xml
2023-01-11T22:18:14.4715456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4715910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4716485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4716944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4717410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94h1db3j
2023-01-11T22:18:14.4717946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94h1db3j/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4718354Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4718550Z 
2023-01-11T22:18:14.4718657Z Running tests...
2023-01-11T22:18:14.4719064Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4720219Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:18:14.4721023Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55861
2023-01-11T22:18:14.4721745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55862
2023-01-11T22:18:14.4722376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4722812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4723391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4723860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4724508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:18:14.4724947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:14.4725522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:14.4725985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:14.4726456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlgzqld2
2023-01-11T22:18:14.4726979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlgzqld2/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4727512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuc5weqm0
2023-01-11T22:18:14.4728045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuc5weqm0/_remote_module_non_scriptable.py
2023-01-11T22:18:14.4728459Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4728868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:14.4729355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:14.4729753Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:14.4730132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:14.4730618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:14.4731277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4731947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:14.4732338Z ok (5.327s)
2023-01-11T22:18:14.4732486Z 
2023-01-11T22:18:14.4732757Z ----------------------------------------------------------------------
2023-01-11T22:18:14.4733084Z Ran 1 test in 5.327s
2023-01-11T22:18:14.4733227Z 
2023-01-11T22:18:14.4733319Z OK
2023-01-11T22:18:14.4733452Z 
2023-01-11T22:18:14.4733576Z Generating XML reports...
2023-01-11T22:18:14.4734216Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221808.xml
2023-01-11T22:18:14.4734603Z 
2023-01-11T22:18:14.4734968Z ##[endgroup]
2023-01-11T22:18:14.4735553Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_ooxwhz4g)
2023-01-11T22:18:14.4735893Z 
2023-01-11T22:18:14.4736136Z Running distributed/test_store ... [2023-01-11 22:18:14.453421]
2023-01-11T22:18:14.4736809Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:14.453686]
2023-01-11T22:20:19.6944919Z 
2023-01-11T22:20:19.6945900Z Expand the folded group to see the log file of distributed/test_store
2023-01-11T22:20:19.6946777Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_bb509_nv)
2023-01-11T22:20:19.6947519Z <unittest.suite.TestSuite tests=[<__main__.FileStoreTest testMethod=test_compare_set>, <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]>
2023-01-11T22:20:19.6948464Z test_compare_set (__main__.FileStoreTest)
2023-01-11T22:20:19.6953033Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest)
2023-01-11T22:20:19.6953736Z test_refcount (__main__.FileStoreTest)
2023-01-11T22:20:19.6954170Z test_set_get (__main__.FileStoreTest)
2023-01-11T22:20:19.6954638Z <unittest.suite.TestSuite tests=[<__main__.HashStoreTest testMethod=test_compare_set>, <__main__.HashStoreTest testMethod=test_set_get>]>
2023-01-11T22:20:19.6955411Z test_compare_set (__main__.HashStoreTest)
2023-01-11T22:20:19.6955996Z test_set_get (__main__.HashStoreTest)
2023-01-11T22:20:19.6956782Z <unittest.suite.TestSuite tests=[<__main__.PrefixFileStoreTest testMethod=test_compare_set>, <__main__.PrefixFileStoreTest testMethod=test_set_get>]>
2023-01-11T22:20:19.6957926Z test_compare_set (__main__.PrefixFileStoreTest)
2023-01-11T22:20:19.6958545Z test_set_get (__main__.PrefixFileStoreTest)
2023-01-11T22:20:19.6959905Z <unittest.suite.TestSuite tests=[<__main__.PrefixStoreTest testMethod=test_get_underlying_store>]>
2023-01-11T22:20:19.6960774Z test_get_underlying_store (__main__.PrefixStoreTest)
2023-01-11T22:20:19.6961661Z <unittest.suite.TestSuite tests=[<__main__.PrefixTCPStoreTest testMethod=test_compare_set>, <__main__.PrefixTCPStoreTest testMethod=test_set_get>]>
2023-01-11T22:20:19.6962515Z test_compare_set (__main__.PrefixTCPStoreTest)
2023-01-11T22:20:19.6963308Z test_set_get (__main__.PrefixTCPStoreTest)
2023-01-11T22:20:19.6964016Z <unittest.suite.TestSuite tests=[<__main__.PythonStoreTest testMethod=test_set_get>]>
2023-01-11T22:20:19.6964653Z test_set_get (__main__.PythonStoreTest)
2023-01-11T22:20:19.6965303Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_nominal>]>
2023-01-11T22:20:19.6965966Z test_nominal (__main__.RendezvousEnvTest)
2023-01-11T22:20:19.6966787Z <unittest.suite.TestSuite tests=[<__main__.RendezvousFileTest testMethod=test_common_errors>, <__main__.RendezvousFileTest testMethod=test_nominal>]>
2023-01-11T22:20:19.6967619Z test_common_errors (__main__.RendezvousFileTest)
2023-01-11T22:20:19.6968205Z test_nominal (__main__.RendezvousFileTest)
2023-01-11T22:20:19.6969402Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTCPTest testMethod=test_common_errors>, <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]>
2023-01-11T22:20:19.6970038Z test_common_errors (__main__.RendezvousTCPTest)
2023-01-11T22:20:19.6970376Z test_dns_timeout (__main__.RendezvousTCPTest)
2023-01-11T22:20:19.6970697Z test_nominal (__main__.RendezvousTCPTest)
2023-01-11T22:20:19.6971037Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2023-01-11T22:20:19.6971532Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTest testMethod=test_unknown_handler>, <__main__.RendezvousTest testMethod=test_url_with_node_params>]>
2023-01-11T22:20:19.6972016Z test_unknown_handler (__main__.RendezvousTest)
2023-01-11T22:20:19.6972352Z test_url_with_node_params (__main__.RendezvousTest)
2023-01-11T22:20:19.6973323Z <unittest.suite.TestSuite tests=[<__main__.TCPStoreTest testMethod=test_address_already_in_use>, <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]>
2023-01-11T22:20:19.6974216Z test_address_already_in_use (__main__.TCPStoreTest)
2023-01-11T22:20:19.6974539Z test_compare_set (__main__.TCPStoreTest)
2023-01-11T22:20:19.6974882Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2023-01-11T22:20:19.6975253Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2023-01-11T22:20:19.6975779Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2023-01-11T22:20:19.6976128Z test_multitenancy (__main__.TCPStoreTest)
2023-01-11T22:20:19.6976449Z test_numkeys_delkeys (__main__.TCPStoreTest)
2023-01-11T22:20:19.6976737Z test_set_get (__main__.TCPStoreTest)
2023-01-11T22:20:19.6977413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.6977862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.6978503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.6978991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.6979222Z 
2023-01-11T22:20:19.6979332Z Running tests...
2023-01-11T22:20:19.6979741Z ----------------------------------------------------------------------
2023-01-11T22:20:19.6980252Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.6980715Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.6981053Z ok (1.619s)
2023-01-11T22:20:19.6981201Z 
2023-01-11T22:20:19.6981448Z ----------------------------------------------------------------------
2023-01-11T22:20:19.6981771Z Ran 1 test in 1.619s
2023-01-11T22:20:19.6981936Z 
2023-01-11T22:20:19.6982029Z OK
2023-01-11T22:20:19.6982162Z 
2023-01-11T22:20:19.6982288Z Generating XML reports...
2023-01-11T22:20:19.6982831Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221818.xml
2023-01-11T22:20:19.6983507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.6983955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.6984509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.6984974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.6985199Z 
2023-01-11T22:20:19.6985307Z Running tests...
2023-01-11T22:20:19.6985704Z ----------------------------------------------------------------------
2023-01-11T22:20:19.6986201Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.6986687Z   test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.6987186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:20:19.6987832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:20:19.6988224Z ok (1.696s)
2023-01-11T22:20:19.6988369Z 
2023-01-11T22:20:19.6988630Z ----------------------------------------------------------------------
2023-01-11T22:20:19.6988955Z Ran 1 test in 1.696s
2023-01-11T22:20:19.6989099Z 
2023-01-11T22:20:19.6989191Z OK
2023-01-11T22:20:19.6989322Z 
2023-01-11T22:20:19.6989446Z Generating XML reports...
2023-01-11T22:20:19.6989994Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221822.xml
2023-01-11T22:20:19.6990648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.6991092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.6991669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.6992130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.6992339Z 
2023-01-11T22:20:19.6992447Z Running tests...
2023-01-11T22:20:19.6992841Z ----------------------------------------------------------------------
2023-01-11T22:20:19.6993445Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.6993883Z   test_refcount (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.6994211Z ok (1.642s)
2023-01-11T22:20:19.6994360Z 
2023-01-11T22:20:19.6994627Z ----------------------------------------------------------------------
2023-01-11T22:20:19.6994948Z Ran 1 test in 1.642s
2023-01-11T22:20:19.6995091Z 
2023-01-11T22:20:19.6995183Z OK
2023-01-11T22:20:19.6995315Z 
2023-01-11T22:20:19.6995438Z Generating XML reports...
2023-01-11T22:20:19.6996044Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221825.xml
2023-01-11T22:20:19.6996710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.6997152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.6997723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.6998194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.6998403Z 
2023-01-11T22:20:19.6998510Z Running tests...
2023-01-11T22:20:19.6998905Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7000053Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7000559Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7000891Z ok (1.643s)
2023-01-11T22:20:19.7001036Z 
2023-01-11T22:20:19.7001311Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7001634Z Ran 1 test in 1.643s
2023-01-11T22:20:19.7001778Z 
2023-01-11T22:20:19.7001871Z OK
2023-01-11T22:20:19.7002003Z 
2023-01-11T22:20:19.7002126Z Generating XML reports...
2023-01-11T22:20:19.7002674Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221829.xml
2023-01-11T22:20:19.7003335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7003779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7004350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7004816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7005026Z 
2023-01-11T22:20:19.7005139Z Running tests...
2023-01-11T22:20:19.7005542Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7006057Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7006497Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7006830Z ok (1.656s)
2023-01-11T22:20:19.7006979Z 
2023-01-11T22:20:19.7007243Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7007548Z Ran 1 test in 1.656s
2023-01-11T22:20:19.7007706Z 
2023-01-11T22:20:19.7007801Z OK
2023-01-11T22:20:19.7007937Z 
2023-01-11T22:20:19.7008061Z Generating XML reports...
2023-01-11T22:20:19.7008607Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111221833.xml
2023-01-11T22:20:19.7009262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7009709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7010281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7010731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7010954Z 
2023-01-11T22:20:19.7011061Z Running tests...
2023-01-11T22:20:19.7011579Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7012096Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7012530Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7012856Z ok (1.638s)
2023-01-11T22:20:19.7013001Z 
2023-01-11T22:20:19.7013263Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7013567Z Ran 1 test in 1.638s
2023-01-11T22:20:19.7013724Z 
2023-01-11T22:20:19.7013816Z OK
2023-01-11T22:20:19.7014021Z 
2023-01-11T22:20:19.7014154Z Generating XML reports...
2023-01-11T22:20:19.7014708Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111221837.xml
2023-01-11T22:20:19.7015352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7015794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7016421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7016872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7017099Z 
2023-01-11T22:20:19.7017207Z Running tests...
2023-01-11T22:20:19.7017604Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7018116Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7018580Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7018927Z ok (1.631s)
2023-01-11T22:20:19.7019074Z 
2023-01-11T22:20:19.7019336Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7019639Z Ran 1 test in 1.631s
2023-01-11T22:20:19.7019796Z 
2023-01-11T22:20:19.7019886Z OK
2023-01-11T22:20:19.7020018Z 
2023-01-11T22:20:19.7020146Z Generating XML reports...
2023-01-11T22:20:19.7020719Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111221841.xml
2023-01-11T22:20:19.7021389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7021830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7022398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7022850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7023077Z 
2023-01-11T22:20:19.7023183Z Running tests...
2023-01-11T22:20:19.7023581Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7024092Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7024543Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7024889Z ok (1.650s)
2023-01-11T22:20:19.7025032Z 
2023-01-11T22:20:19.7025290Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7025594Z Ran 1 test in 1.651s
2023-01-11T22:20:19.7025751Z 
2023-01-11T22:20:19.7025843Z OK
2023-01-11T22:20:19.7025974Z 
2023-01-11T22:20:19.7026097Z Generating XML reports...
2023-01-11T22:20:19.7026665Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111221845.xml
2023-01-11T22:20:19.7027335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7027784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7028355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7028802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7029106Z 
2023-01-11T22:20:19.7029214Z Running tests...
2023-01-11T22:20:19.7029614Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7030130Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7030540Z   test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.003s)
2023-01-11T22:20:19.7030761Z 
2023-01-11T22:20:19.7031017Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7031381Z Ran 1 test in 0.003s
2023-01-11T22:20:19.7031549Z 
2023-01-11T22:20:19.7031624Z OK
2023-01-11T22:20:19.7031757Z 
2023-01-11T22:20:19.7031880Z Generating XML reports...
2023-01-11T22:20:19.7032440Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20230111221849.xml
2023-01-11T22:20:19.7033109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7033540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7034109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7034570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7034796Z 
2023-01-11T22:20:19.7034888Z Running tests...
2023-01-11T22:20:19.7035284Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7035804Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7036275Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7036605Z ok (1.651s)
2023-01-11T22:20:19.7036750Z 
2023-01-11T22:20:19.7037009Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7037325Z Ran 1 test in 1.652s
2023-01-11T22:20:19.7037495Z 
2023-01-11T22:20:19.7037564Z OK
2023-01-11T22:20:19.7037697Z 
2023-01-11T22:20:19.7037821Z Generating XML reports...
2023-01-11T22:20:19.7038387Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111221851.xml
2023-01-11T22:20:19.7039069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7040093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7040705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7041171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7041399Z 
2023-01-11T22:20:19.7041490Z Running tests...
2023-01-11T22:20:19.7041885Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7042397Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7042867Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7043189Z ok (1.641s)
2023-01-11T22:20:19.7043335Z 
2023-01-11T22:20:19.7043596Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7043918Z Ran 1 test in 1.642s
2023-01-11T22:20:19.7044077Z 
2023-01-11T22:20:19.7044304Z OK
2023-01-11T22:20:19.7044437Z 
2023-01-11T22:20:19.7044562Z Generating XML reports...
2023-01-11T22:20:19.7045138Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111221855.xml
2023-01-11T22:20:19.7045821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7046248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7046819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7047395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7047620Z 
2023-01-11T22:20:19.7047710Z Running tests...
2023-01-11T22:20:19.7048113Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7048627Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7049088Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7049403Z ok (1.628s)
2023-01-11T22:20:19.7049547Z 
2023-01-11T22:20:19.7049879Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7050210Z Ran 1 test in 1.628s
2023-01-11T22:20:19.7050369Z 
2023-01-11T22:20:19.7050444Z OK
2023-01-11T22:20:19.7050575Z 
2023-01-11T22:20:19.7050697Z Generating XML reports...
2023-01-11T22:20:19.7051254Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20230111221859.xml
2023-01-11T22:20:19.7051930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7052357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7052925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7053390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7053617Z 
2023-01-11T22:20:19.7053709Z Running tests...
2023-01-11T22:20:19.7054110Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7054621Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7055081Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7055401Z ok (1.647s)
2023-01-11T22:20:19.7055545Z 
2023-01-11T22:20:19.7055805Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7056130Z Ran 1 test in 1.647s
2023-01-11T22:20:19.7056289Z 
2023-01-11T22:20:19.7056366Z OK
2023-01-11T22:20:19.7056499Z 
2023-01-11T22:20:19.7056623Z Generating XML reports...
2023-01-11T22:20:19.7057182Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20230111221903.xml
2023-01-11T22:20:19.7057842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7058291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7058862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7059322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7059546Z 
2023-01-11T22:20:19.7059637Z Running tests...
2023-01-11T22:20:19.7060036Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7060554Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7061022Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7061347Z ok (1.621s)
2023-01-11T22:20:19.7061494Z 
2023-01-11T22:20:19.7061754Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7062072Z Ran 1 test in 1.621s
2023-01-11T22:20:19.7062214Z 
2023-01-11T22:20:19.7062308Z OK
2023-01-11T22:20:19.7062440Z 
2023-01-11T22:20:19.7062566Z Generating XML reports...
2023-01-11T22:20:19.7063131Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111221906.xml
2023-01-11T22:20:19.7063795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7064240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7064903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7065370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7065596Z 
2023-01-11T22:20:19.7065687Z Running tests...
2023-01-11T22:20:19.7066089Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7066604Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7067105Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7067454Z ok (1.637s)
2023-01-11T22:20:19.7067599Z 
2023-01-11T22:20:19.7067859Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7068177Z Ran 1 test in 1.638s
2023-01-11T22:20:19.7068317Z 
2023-01-11T22:20:19.7068408Z OK
2023-01-11T22:20:19.7068538Z 
2023-01-11T22:20:19.7068663Z Generating XML reports...
2023-01-11T22:20:19.7069231Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111221910.xml
2023-01-11T22:20:19.7069894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7070333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7070901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7071363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7071592Z 
2023-01-11T22:20:19.7071685Z Running tests...
2023-01-11T22:20:19.7072082Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7072593Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7073046Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7073397Z ok (1.628s)
2023-01-11T22:20:19.7073542Z 
2023-01-11T22:20:19.7073802Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7074121Z Ran 1 test in 1.628s
2023-01-11T22:20:19.7074261Z 
2023-01-11T22:20:19.7074353Z OK
2023-01-11T22:20:19.7074484Z 
2023-01-11T22:20:19.7074606Z Generating XML reports...
2023-01-11T22:20:19.7075162Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221914.xml
2023-01-11T22:20:19.7075826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7076267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7076837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7077298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7077506Z 
2023-01-11T22:20:19.7077618Z Running tests...
2023-01-11T22:20:19.7078015Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7078528Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7078976Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7080235Z [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2023-01-11T22:20:19.7080777Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2023-01-11T22:20:19.7081126Z ok (1.647s)
2023-01-11T22:20:19.7081271Z 
2023-01-11T22:20:19.7081532Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7081855Z Ran 1 test in 1.648s
2023-01-11T22:20:19.7082013Z 
2023-01-11T22:20:19.7082106Z OK
2023-01-11T22:20:19.7082238Z 
2023-01-11T22:20:19.7082451Z Generating XML reports...
2023-01-11T22:20:19.7083025Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221918.xml
2023-01-11T22:20:19.7083702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7084146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7084700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7085233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7085470Z 
2023-01-11T22:20:19.7085578Z Running tests...
2023-01-11T22:20:19.7085965Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7086478Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7086940Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7087284Z ok (1.633s)
2023-01-11T22:20:19.7087413Z 
2023-01-11T22:20:19.7087674Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7087991Z Ran 1 test in 1.633s
2023-01-11T22:20:19.7088149Z 
2023-01-11T22:20:19.7088243Z OK
2023-01-11T22:20:19.7088374Z 
2023-01-11T22:20:19.7088480Z Generating XML reports...
2023-01-11T22:20:19.7089036Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221922.xml
2023-01-11T22:20:19.7089717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7090163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7090715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7091176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7091405Z 
2023-01-11T22:20:19.7091513Z Running tests...
2023-01-11T22:20:19.7091895Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7092407Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7092886Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7093242Z ok (11.886s)
2023-01-11T22:20:19.7093372Z 
2023-01-11T22:20:19.7093632Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7093957Z Ran 1 test in 11.886s
2023-01-11T22:20:19.7094115Z 
2023-01-11T22:20:19.7094208Z OK
2023-01-11T22:20:19.7094338Z 
2023-01-11T22:20:19.7094445Z Generating XML reports...
2023-01-11T22:20:19.7095006Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221926.xml
2023-01-11T22:20:19.7095682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7096129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7096679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7097140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7097364Z 
2023-01-11T22:20:19.7097472Z Running tests...
2023-01-11T22:20:19.7097854Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7098374Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7098836Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7099178Z ok (1.642s)
2023-01-11T22:20:19.7099306Z 
2023-01-11T22:20:19.7099566Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7099962Z Ran 1 test in 1.642s
2023-01-11T22:20:19.7100120Z 
2023-01-11T22:20:19.7100213Z OK
2023-01-11T22:20:19.7100344Z 
2023-01-11T22:20:19.7100450Z Generating XML reports...
2023-01-11T22:20:19.7101005Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111221940.xml
2023-01-11T22:20:19.7101672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7102116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7102721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7103196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7103422Z 
2023-01-11T22:20:19.7103530Z Running tests...
2023-01-11T22:20:19.7103915Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7104427Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7104902Z   test_url_with_node_params (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7105245Z ok (1.640s)
2023-01-11T22:20:19.7105373Z 
2023-01-11T22:20:19.7105635Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7105954Z Ran 1 test in 1.641s
2023-01-11T22:20:19.7106112Z 
2023-01-11T22:20:19.7106204Z OK
2023-01-11T22:20:19.7106335Z 
2023-01-11T22:20:19.7106440Z Generating XML reports...
2023-01-11T22:20:19.7106995Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111221944.xml
2023-01-11T22:20:19.7107662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7108103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7108651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7109122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7109349Z 
2023-01-11T22:20:19.7109458Z Running tests...
2023-01-11T22:20:19.7109840Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7110353Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7110823Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7111435Z [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:33139 (errno: 98 - Address already in use).
2023-01-11T22:20:19.7112012Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:33139 (errno: 98 - Address already in use).
2023-01-11T22:20:19.7112470Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2023-01-11T22:20:19.7112799Z ok (1.637s)
2023-01-11T22:20:19.7112948Z 
2023-01-11T22:20:19.7113192Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7113514Z Ran 1 test in 1.638s
2023-01-11T22:20:19.7113672Z 
2023-01-11T22:20:19.7113762Z OK
2023-01-11T22:20:19.7113893Z 
2023-01-11T22:20:19.7114017Z Generating XML reports...
2023-01-11T22:20:19.7114543Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221947.xml
2023-01-11T22:20:19.7115209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7115654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7116206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7116730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7116960Z 
2023-01-11T22:20:19.7117067Z Running tests...
2023-01-11T22:20:19.7117547Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7118044Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7118501Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7118833Z ok (1.644s)
2023-01-11T22:20:19.7118976Z 
2023-01-11T22:20:19.7119218Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7120095Z Ran 1 test in 1.644s
2023-01-11T22:20:19.7120260Z 
2023-01-11T22:20:19.7120449Z OK
2023-01-11T22:20:19.7120594Z 
2023-01-11T22:20:19.7120719Z Generating XML reports...
2023-01-11T22:20:19.7121267Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221951.xml
2023-01-11T22:20:19.7121930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7122375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7122931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7123401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7123629Z 
2023-01-11T22:20:19.7123736Z Running tests...
2023-01-11T22:20:19.7124130Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7124624Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7125113Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7125613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:20:19.7126251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:20:19.7126648Z ok (1.728s)
2023-01-11T22:20:19.7126793Z 
2023-01-11T22:20:19.7127055Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7127374Z Ran 1 test in 1.728s
2023-01-11T22:20:19.7127516Z 
2023-01-11T22:20:19.7127609Z OK
2023-01-11T22:20:19.7127740Z 
2023-01-11T22:20:19.7127867Z Generating XML reports...
2023-01-11T22:20:19.7128411Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221955.xml
2023-01-11T22:20:19.7129059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7129505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7130077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7130538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7130750Z 
2023-01-11T22:20:19.7130863Z Running tests...
2023-01-11T22:20:19.7131259Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7131776Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7132250Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7132609Z ok (1.654s)
2023-01-11T22:20:19.7132755Z 
2023-01-11T22:20:19.7133016Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7133337Z Ran 1 test in 1.654s
2023-01-11T22:20:19.7133485Z 
2023-01-11T22:20:19.7133578Z OK
2023-01-11T22:20:19.7133711Z 
2023-01-11T22:20:19.7133834Z Generating XML reports...
2023-01-11T22:20:19.7134380Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221959.xml
2023-01-11T22:20:19.7135025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7135592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7136164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7136628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7136837Z 
2023-01-11T22:20:19.7136946Z Running tests...
2023-01-11T22:20:19.7137340Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7137908Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7138394Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7138752Z ok (1.636s)
2023-01-11T22:20:19.7138897Z 
2023-01-11T22:20:19.7139158Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7139481Z Ran 1 test in 1.636s
2023-01-11T22:20:19.7139626Z 
2023-01-11T22:20:19.7139717Z OK
2023-01-11T22:20:19.7139848Z 
2023-01-11T22:20:19.7139973Z Generating XML reports...
2023-01-11T22:20:19.7140516Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222003.xml
2023-01-11T22:20:19.7141163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7141607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7142180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7142641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7142849Z 
2023-01-11T22:20:19.7142958Z Running tests...
2023-01-11T22:20:19.7143353Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7143865Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7144314Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7144650Z ok (1.641s)
2023-01-11T22:20:19.7144794Z 
2023-01-11T22:20:19.7145055Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7145373Z Ran 1 test in 1.641s
2023-01-11T22:20:19.7145514Z 
2023-01-11T22:20:19.7145605Z OK
2023-01-11T22:20:19.7145737Z 
2023-01-11T22:20:19.7145859Z Generating XML reports...
2023-01-11T22:20:19.7146403Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222007.xml
2023-01-11T22:20:19.7147053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7147500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7148071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7148539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7148747Z 
2023-01-11T22:20:19.7148854Z Running tests...
2023-01-11T22:20:19.7149247Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7149759Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7150205Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7150541Z ok (3.654s)
2023-01-11T22:20:19.7150687Z 
2023-01-11T22:20:19.7150954Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7151272Z Ran 1 test in 3.655s
2023-01-11T22:20:19.7151413Z 
2023-01-11T22:20:19.7151505Z OK
2023-01-11T22:20:19.7151636Z 
2023-01-11T22:20:19.7151758Z Generating XML reports...
2023-01-11T22:20:19.7152300Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222011.xml
2023-01-11T22:20:19.7153025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:20:19.7153466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:19.7154040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:19.7154507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:19.7154717Z 
2023-01-11T22:20:19.7154826Z Running tests...
2023-01-11T22:20:19.7155276Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7155802Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T22:20:19.7156234Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:19.7156557Z ok (1.633s)
2023-01-11T22:20:19.7156703Z 
2023-01-11T22:20:19.7156962Z ----------------------------------------------------------------------
2023-01-11T22:20:19.7157269Z Ran 1 test in 1.634s
2023-01-11T22:20:19.7157430Z 
2023-01-11T22:20:19.7157521Z OK
2023-01-11T22:20:19.7157653Z 
2023-01-11T22:20:19.7157776Z Generating XML reports...
2023-01-11T22:20:19.7158319Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222017.xml
2023-01-11T22:20:19.7158619Z 
2023-01-11T22:20:19.7158963Z ##[endgroup]
2023-01-11T22:20:19.7159987Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_bb509_nv)
2023-01-11T22:20:19.7160327Z 
2023-01-11T22:20:19.7160610Z Running distributed/_tensor/test_tensor_ops ... [2023-01-11 22:20:19.695131]
2023-01-11T22:20:19.7161273Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:19.695417]
2023-01-11T22:21:34.2144619Z 
2023-01-11T22:21:34.2145386Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops
2023-01-11T22:21:34.2146349Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_64kphrja)
2023-01-11T22:21:34.2146705Z 
2023-01-11T22:21:34.2146814Z Running tests...
2023-01-11T22:21:34.2153370Z ----------------------------------------------------------------------
2023-01-11T22:21:34.2154699Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops
2023-01-11T22:21:34.2155400Z   test_aten_contiguous (__main__.DistTensorOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:34.2155870Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57125
2023-01-11T22:21:34.2156789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57126
2023-01-11T22:21:34.2157815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2158681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2160173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2160871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2161434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2161947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2162496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2162984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2163411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2163903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2164657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2165487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2166461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2168044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2168796Z ok (5.654s)
2023-01-11T22:21:34.2169518Z   test_clone (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57206
2023-01-11T22:21:34.2170398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57207
2023-01-11T22:21:34.2171521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2172355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2172938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2173698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2174648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2175569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2176676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2177548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2178275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2179090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2179892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2180358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2181008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2181703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2182095Z ok (3.910s)
2023-01-11T22:21:34.2182499Z   test_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57287
2023-01-11T22:21:34.2183008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57288
2023-01-11T22:21:34.2183619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2184050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2184629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2185092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2185670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2186096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2186665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2187123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2187560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2188155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2188638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2189123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2189762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2190514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2190915Z ok (3.911s)
2023-01-11T22:21:34.2191326Z   test_detach (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57370
2023-01-11T22:21:34.2191810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57371
2023-01-11T22:21:34.2192427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2192871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2193445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2193895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2194474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2194915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2195468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2195930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2196363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2196854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2197315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2197797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2198455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2199123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2199879Z ok (3.910s)
2023-01-11T22:21:34.2200292Z   test_empty_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57451
2023-01-11T22:21:34.2200802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57452
2023-01-11T22:21:34.2201404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2201850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2202422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2202881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2203445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2203884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2204460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2204896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2205458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2205946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2206425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2206889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2207620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2208331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2208716Z ok (3.910s)
2023-01-11T22:21:34.2209116Z   test_fill_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57530
2023-01-11T22:21:34.2209629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57531
2023-01-11T22:21:34.2210239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2210668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2211239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2211704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2212287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2212711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2213279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2213739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2214159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2214649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2215127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2215610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2216248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2216931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2217316Z ok (3.910s)
2023-01-11T22:21:34.2217750Z   test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57609
2023-01-11T22:21:34.2218262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57610
2023-01-11T22:21:34.2218868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2219317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2219865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2220335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2220914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2221352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2221904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2222437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2222921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2223391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2223874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2224354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2225071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2225749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2226143Z ok (3.810s)
2023-01-11T22:21:34.2226554Z   test_full_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57688
2023-01-11T22:21:34.2227064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57689
2023-01-11T22:21:34.2227654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2228094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2228665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2229117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2229700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2230135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2230706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2231152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2231583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2232069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2232531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2233011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2233666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2234348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2234717Z ok (3.910s)
2023-01-11T22:21:34.2235124Z   test_index (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57767
2023-01-11T22:21:34.2235631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57768
2023-01-11T22:21:34.2236237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2236664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2237235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2237703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2238264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2238701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2239269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2240125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2240543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2241029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2241515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2242072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2242731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2243418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2243813Z ok (11.928s)
2023-01-11T22:21:34.2244218Z   test_inplace_op (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57878
2023-01-11T22:21:34.2244809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57879
2023-01-11T22:21:34.2245405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2245852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2246424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2246891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2247448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2247888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2248458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2248909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2249343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2249830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2250315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2250776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2251429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2252114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2252507Z ok (3.911s)
2023-01-11T22:21:34.2252899Z   test_ones_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57959
2023-01-11T22:21:34.2253405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57960
2023-01-11T22:21:34.2254007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2254432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2255006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2255470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2256043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2256460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2257126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2257587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2258003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2258487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2259024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2259521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2260161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2260841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2261233Z ok (3.910s)
2023-01-11T22:21:34.2261659Z   test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58038
2023-01-11T22:21:34.2262161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58039
2023-01-11T22:21:34.2262764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2263209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2263765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2264231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2264807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2265246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2265804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2266266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2266699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2267184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2267648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2268128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2268786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2269448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2269841Z ok (3.910s)
2023-01-11T22:21:34.2270260Z   test_op_out_variant (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58117
2023-01-11T22:21:34.2270770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58118
2023-01-11T22:21:34.2271355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2271799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2272372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2272817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2273395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2273926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2274499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2274947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2275383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2275866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2276400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2276871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2277531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2278224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2278595Z ok (3.910s)
2023-01-11T22:21:34.2279008Z   test_zero_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58198
2023-01-11T22:21:34.2279900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58199
2023-01-11T22:21:34.2280506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2280944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2281520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2281984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2282562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2282990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2283559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2284016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2284431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2284920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2285400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2285876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2286510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2287198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2287590Z ok (3.910s)
2023-01-11T22:21:34.2287985Z   test_zeros_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58277
2023-01-11T22:21:34.2288491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58278
2023-01-11T22:21:34.2289099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2289547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2290104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2290567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2291141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2291683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2292239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2292699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2293138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2293675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2294171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2294652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2295303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2295975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2296363Z ok (3.910s)
2023-01-11T22:21:34.2296790Z   test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58356
2023-01-11T22:21:34.2297290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58357
2023-01-11T22:21:34.2297898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2298347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2298919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2299368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2299946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:21:34.2300383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:34.2300951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:34.2301396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:34.2301832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:34.2302315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:34.2302778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:34.2303256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:34.2303913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2304594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:34.2304966Z ok (3.910s)
2023-01-11T22:21:34.2305115Z 
2023-01-11T22:21:34.2305382Z ----------------------------------------------------------------------
2023-01-11T22:21:34.2305710Z Ran 16 tests in 72.228s
2023-01-11T22:21:34.2305871Z 
2023-01-11T22:21:34.2305946Z OK
2023-01-11T22:21:34.2306079Z 
2023-01-11T22:21:34.2306209Z Generating XML reports...
2023-01-11T22:21:34.2306801Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20230111222021.xml
2023-01-11T22:21:34.2307147Z 
2023-01-11T22:21:34.2307611Z ##[endgroup]
2023-01-11T22:21:34.2308197Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_64kphrja)
2023-01-11T22:21:34.2308630Z 
2023-01-11T22:21:34.2308911Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2023-01-11 22:21:34.214826]
2023-01-11T22:21:34.2309601Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:21:34.215096]
2023-01-11T22:23:35.3902013Z 
2023-01-11T22:23:35.3902522Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T22:23:35.3907651Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_4f9sjntz)
2023-01-11T22:23:35.3908429Z 
2023-01-11T22:23:35.3909492Z Running tests...
2023-01-11T22:23:35.3910544Z ----------------------------------------------------------------------
2023-01-11T22:23:35.3911176Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2023-01-11T22:23:35.3911802Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:23:35.3912364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58470
2023-01-11T22:23:35.3912818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58471
2023-01-11T22:23:35.3913426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3913894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3914484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3914935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3915769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3916231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3916811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3917270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3917721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3922267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3923032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3923982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3924550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3925022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3925376Z dist init r=1, world=2
2023-01-11T22:23:35.3925629Z dist init r=0, world=2
2023-01-11T22:23:35.3925868Z ok (6.037s)
2023-01-11T22:23:35.3926361Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58553
2023-01-11T22:23:35.3926969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58554
2023-01-11T22:23:35.3927743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3928202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3928770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3929461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3930056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3930494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3931040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3931523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3932078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3932583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3933309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3933985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3934517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3934982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3935304Z dist init r=0, world=2
2023-01-11T22:23:35.3935554Z dist init r=1, world=2
2023-01-11T22:23:35.3935796Z ok (4.411s)
2023-01-11T22:23:35.3936297Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58636
2023-01-11T22:23:35.3936899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58637
2023-01-11T22:23:35.3937511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3937946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3938527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3938990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3939566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3939988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3940560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3941020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3941471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3941944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3942606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3943293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3943792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3944260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3944611Z dist init r=1, world=2
2023-01-11T22:23:35.3944862Z dist init r=0, world=2
2023-01-11T22:23:35.3945084Z ok (4.411s)
2023-01-11T22:23:35.3945597Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58719
2023-01-11T22:23:35.3946195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58720
2023-01-11T22:23:35.3946862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3947309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3947886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3948348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3948956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3949414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3949990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3950450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3950889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3951382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3952037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3952706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3953228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3953694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3954048Z dist init r=1, world=2
2023-01-11T22:23:35.3954281Z dist init r=0, world=2
2023-01-11T22:23:35.3954519Z ok (4.411s)
2023-01-11T22:23:35.3955028Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58802
2023-01-11T22:23:35.3955610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58803
2023-01-11T22:23:35.3956220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3956670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3957246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3957689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3958267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3958704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3959255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3960029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3960482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3960974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3961623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3962310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3962830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3963300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3963732Z dist init r=0, world=2
2023-01-11T22:23:35.3963983Z dist init r=1, world=2
2023-01-11T22:23:35.3964223Z ok (4.411s)
2023-01-11T22:23:35.3964718Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58885
2023-01-11T22:23:35.3965317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58886
2023-01-11T22:23:35.3965992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3966445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3967004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3967471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3968047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3968471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3969041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3969501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3969948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3970428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3971084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3971764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3972278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3972734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3973080Z dist init r=0, world=2
2023-01-11T22:23:35.3973327Z dist init r=1, world=2
2023-01-11T22:23:35.3973548Z ok (4.511s)
2023-01-11T22:23:35.3973964Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.3974712Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58968
2023-01-11T22:23:35.3975251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58969
2023-01-11T22:23:35.3975835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3976283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3976862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3977308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3977888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3978330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3978903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3979349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3979803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3980296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3980996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3981681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3982195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3982659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3982999Z dist init r=1, world=2
2023-01-11T22:23:35.3983326Z dist init r=0, world=2
2023-01-11T22:23:35.3983574Z ok (4.413s)
2023-01-11T22:23:35.3983969Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.3984711Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59051
2023-01-11T22:23:35.3985252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59052
2023-01-11T22:23:35.3985854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3986283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3986855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3987319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3987901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3988320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3988898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3989357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3989791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.3990279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.3990929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3991612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.3992113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.3992580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.3992937Z dist init r=1, world=2
2023-01-11T22:23:35.3993170Z dist init r=0, world=2
2023-01-11T22:23:35.3993408Z ok (4.412s)
2023-01-11T22:23:35.3993826Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.3994573Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59134
2023-01-11T22:23:35.3995090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59135
2023-01-11T22:23:35.3995691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3996139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3996692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3997158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.3997733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.3998237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.3998792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.3999254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4000068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4000706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4001369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4002051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4002571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4003053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4003405Z dist init r=1, world=2
2023-01-11T22:23:35.4003656Z dist init r=0, world=2
2023-01-11T22:23:35.4003896Z ok (4.512s)
2023-01-11T22:23:35.4004330Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4005076Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59217
2023-01-11T22:23:35.4005593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59218
2023-01-11T22:23:35.4006183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4006631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4007205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4007673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4008232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4008674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4009244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4009710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4010139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4010628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4011283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4011956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4012473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4012943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4013528Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4013897Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4014432Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4014816Z     _check_comm_hook(
2023-01-11T22:23:35.4015306Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4015858Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4016413Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4016790Z     traceback.print_stack()
2023-01-11T22:23:35.4017266Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4017645Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4018219Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4018593Z     _check_comm_hook(
2023-01-11T22:23:35.4019101Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4019569Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4020118Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4020480Z     traceback.print_stack()
2023-01-11T22:23:35.4020974Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4021352Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4021857Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4022238Z     _check_comm_hook(
2023-01-11T22:23:35.4022796Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4023165Z     p_assert(
2023-01-11T22:23:35.4023616Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4023992Z     traceback.print_stack()
2023-01-11T22:23:35.4024483Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4024847Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4025373Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4025755Z     _check_comm_hook(
2023-01-11T22:23:35.4026237Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4026607Z     p_assert(
2023-01-11T22:23:35.4027076Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4027452Z     traceback.print_stack()
2023-01-11T22:23:35.4027702Z dist init r=1, world=2
2023-01-11T22:23:35.4027984Z Communication hook should not be `None`
2023-01-11T22:23:35.4028411Z Communication hook state should not be `None`
2023-01-11T22:23:35.4028681Z dist init r=0, world=2
2023-01-11T22:23:35.4029042Z Communication hook should not be `None`
2023-01-11T22:23:35.4029366Z Communication hook state should not be `None`
2023-01-11T22:23:35.4029630Z ok (4.412s)
2023-01-11T22:23:35.4030081Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4030823Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59300
2023-01-11T22:23:35.4031345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59301
2023-01-11T22:23:35.4031937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4032381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4032998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4033449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4034110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4034550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4035121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4035566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4036066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4036563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4037204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4037885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4038411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4038879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4039755Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4040136Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4040675Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4041060Z     _check_comm_hook(
2023-01-11T22:23:35.4041550Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4042018Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4042572Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4042936Z     traceback.print_stack()
2023-01-11T22:23:35.4043432Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4043802Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4061319Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4061734Z     _check_comm_hook(
2023-01-11T22:23:35.4062243Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4062715Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4063276Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4063657Z     traceback.print_stack()
2023-01-11T22:23:35.4064140Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4064529Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4065018Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4065374Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4065898Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4066278Z     _check_comm_hook(
2023-01-11T22:23:35.4066784Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4067138Z     p_assert(
2023-01-11T22:23:35.4067635Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4068010Z     _check_comm_hook(
2023-01-11T22:23:35.4068468Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4069002Z     traceback.print_stack()
2023-01-11T22:23:35.4069524Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4069888Z     p_assert(
2023-01-11T22:23:35.4070330Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4070699Z     traceback.print_stack()
2023-01-11T22:23:35.4070965Z dist init r=1, world=2
2023-01-11T22:23:35.4071228Z Communication hook should not be `None`
2023-01-11T22:23:35.4071625Z Communication hook state should not be `None`
2023-01-11T22:23:35.4071922Z dist init r=0, world=2
2023-01-11T22:23:35.4072184Z Communication hook should not be `None`
2023-01-11T22:23:35.4072505Z Communication hook state should not be `None`
2023-01-11T22:23:35.4072779Z ok (4.512s)
2023-01-11T22:23:35.4073217Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4073975Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59383
2023-01-11T22:23:35.4074491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59384
2023-01-11T22:23:35.4075091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4075518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4076093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4076559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4077115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4077552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4078119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4078578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4079011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4079857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4080525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4081207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4081707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4082173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4082759Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4083123Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4083651Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4084035Z     _check_comm_hook(
2023-01-11T22:23:35.4084539Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4084988Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4085538Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4085911Z     traceback.print_stack()
2023-01-11T22:23:35.4086386Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4086855Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4087379Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4087758Z     _check_comm_hook(
2023-01-11T22:23:35.4088399Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4088863Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4089483Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4089869Z     traceback.print_stack()
2023-01-11T22:23:35.4090347Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4090718Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4091241Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4091611Z     _check_comm_hook(
2023-01-11T22:23:35.4092109Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4092467Z     p_assert(
2023-01-11T22:23:35.4092913Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4093277Z     traceback.print_stack()
2023-01-11T22:23:35.4093761Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4094123Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4094621Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4094984Z     _check_comm_hook(
2023-01-11T22:23:35.4095474Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4095827Z     p_assert(
2023-01-11T22:23:35.4096286Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4096658Z     traceback.print_stack()
2023-01-11T22:23:35.4096902Z dist init r=0, world=2
2023-01-11T22:23:35.4097181Z Communication hook should not be `None`
2023-01-11T22:23:35.4097504Z Communication hook state should not be `None`
2023-01-11T22:23:35.4097791Z dist init r=1, world=2
2023-01-11T22:23:35.4098052Z Communication hook should not be `None`
2023-01-11T22:23:35.4098376Z Communication hook state should not be `None`
2023-01-11T22:23:35.4098652Z ok (4.412s)
2023-01-11T22:23:35.4099085Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4099828Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59466
2023-01-11T22:23:35.4100347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59467
2023-01-11T22:23:35.4100935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4101379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4101950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4102415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4102974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4103415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4103983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4104519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4104951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4105442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4106097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4106813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4107333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4107798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4108383Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4108752Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4109278Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4109659Z     _check_comm_hook(
2023-01-11T22:23:35.4110148Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4110612Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4111169Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4111545Z     traceback.print_stack()
2023-01-11T22:23:35.4112017Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4112394Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4112914Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4113282Z     _check_comm_hook(
2023-01-11T22:23:35.4113783Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4114247Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4114794Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4115147Z     traceback.print_stack()
2023-01-11T22:23:35.4115639Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4116009Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4116511Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4116890Z     _check_comm_hook(
2023-01-11T22:23:35.4117388Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4117759Z     p_assert(
2023-01-11T22:23:35.4118209Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4118583Z     traceback.print_stack()
2023-01-11T22:23:35.4119068Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4119688Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4120228Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4120604Z     _check_comm_hook(
2023-01-11T22:23:35.4121087Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4121450Z     p_assert(
2023-01-11T22:23:35.4121911Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4122368Z     traceback.print_stack()
2023-01-11T22:23:35.4122614Z dist init r=1, world=2
2023-01-11T22:23:35.4122894Z Communication hook should not be `None`
2023-01-11T22:23:35.4123213Z Communication hook state should not be `None`
2023-01-11T22:23:35.4123483Z dist init r=0, world=2
2023-01-11T22:23:35.4123757Z Communication hook should not be `None`
2023-01-11T22:23:35.4124075Z Communication hook state should not be `None`
2023-01-11T22:23:35.4124331Z ok (4.412s)
2023-01-11T22:23:35.4124834Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4125582Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59549
2023-01-11T22:23:35.4126096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59550
2023-01-11T22:23:35.4126679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4127124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4127692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4128137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4128707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4129149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4129714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4130155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4130604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4131103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4131751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4132417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4132924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4133437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4134004Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4134381Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4134904Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4135288Z     _check_comm_hook(
2023-01-11T22:23:35.4135772Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4136237Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4136788Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4137144Z     traceback.print_stack()
2023-01-11T22:23:35.4137639Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4138015Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4138535Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4138900Z     _check_comm_hook(
2023-01-11T22:23:35.4139398Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4139927Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4140466Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4140836Z     traceback.print_stack()
2023-01-11T22:23:35.4141323Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4141697Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4142248Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4142636Z     _check_comm_hook(
2023-01-11T22:23:35.4143137Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4143483Z     p_assert(
2023-01-11T22:23:35.4143944Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4144323Z     traceback.print_stack()
2023-01-11T22:23:35.4144796Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4145167Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4145687Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4146064Z     _check_comm_hook(
2023-01-11T22:23:35.4146551Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4146915Z     p_assert(
2023-01-11T22:23:35.4147372Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4147726Z     traceback.print_stack()
2023-01-11T22:23:35.4147983Z dist init r=0, world=2
2023-01-11T22:23:35.4148259Z Communication hook should not be `None`
2023-01-11T22:23:35.4148566Z Communication hook state should not be `None`
2023-01-11T22:23:35.4148847Z dist init r=1, world=2
2023-01-11T22:23:35.4149122Z Communication hook should not be `None`
2023-01-11T22:23:35.4149425Z Communication hook state should not be `None`
2023-01-11T22:23:35.4149696Z ok (4.512s)
2023-01-11T22:23:35.4150146Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4150890Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59632
2023-01-11T22:23:35.4151389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59633
2023-01-11T22:23:35.4151987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4152429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4152996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4153447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4154016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4154447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4154995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4155449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4155903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4156559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4157072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4157778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4158288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4158753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4159321Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4160092Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4160598Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4160953Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4161469Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4161853Z     _check_comm_hook(
2023-01-11T22:23:35.4162341Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4162805Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4163395Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4163778Z     _check_comm_hook(
2023-01-11T22:23:35.4164237Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4164610Z     traceback.print_stack()
2023-01-11T22:23:35.4165130Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:23:35.4165590Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:23:35.4166123Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4166500Z     traceback.print_stack()
2023-01-11T22:23:35.4166988Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4167345Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4167828Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:23:35.4168194Z     return func(*args, **kwargs)
2023-01-11T22:23:35.4168704Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4169084Z     _check_comm_hook(
2023-01-11T22:23:35.4169585Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4169952Z     p_assert(
2023-01-11T22:23:35.4170436Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:23:35.4170820Z     _check_comm_hook(
2023-01-11T22:23:35.4171291Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4171649Z     traceback.print_stack()
2023-01-11T22:23:35.4172168Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:23:35.4172534Z     p_assert(
2023-01-11T22:23:35.4172998Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:23:35.4173353Z     traceback.print_stack()
2023-01-11T22:23:35.4173616Z dist init r=0, world=2
2023-01-11T22:23:35.4173897Z Communication hook should not be `None`
2023-01-11T22:23:35.4174204Z Communication hook state should not be `None`
2023-01-11T22:23:35.4174491Z dist init r=1, world=2
2023-01-11T22:23:35.4174768Z Communication hook should not be `None`
2023-01-11T22:23:35.4175070Z Communication hook state should not be `None`
2023-01-11T22:23:35.4175426Z ok (4.412s)
2023-01-11T22:23:35.4175939Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59715
2023-01-11T22:23:35.4176523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59716
2023-01-11T22:23:35.4177138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4177642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4178223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4178673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4179246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4179687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4180255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4180698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4181146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4181643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4182276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4182957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4183473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4183940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4184273Z dist init r=0, world=2
2023-01-11T22:23:35.4184520Z dist init r=1, world=2
2023-01-11T22:23:35.4184756Z ok (4.412s)
2023-01-11T22:23:35.4185248Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59798
2023-01-11T22:23:35.4185848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59799
2023-01-11T22:23:35.4186452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4186892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4187444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4187909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4188480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4188899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4189459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4189912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4190360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4190839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4191482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4192225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4192739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4193186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4193529Z dist init r=1, world=2
2023-01-11T22:23:35.4193777Z dist init r=0, world=2
2023-01-11T22:23:35.4193999Z ok (4.512s)
2023-01-11T22:23:35.4194569Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59881
2023-01-11T22:23:35.4195174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59882
2023-01-11T22:23:35.4195778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4196210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4196774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4197232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4197785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4198220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4198785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4199240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4199876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4200375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4201035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4201714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4202216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4202683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4203039Z dist init r=1, world=2
2023-01-11T22:23:35.4203270Z dist init r=0, world=2
2023-01-11T22:23:35.4203505Z ok (4.511s)
2023-01-11T22:23:35.4204015Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59964
2023-01-11T22:23:35.4204608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59965
2023-01-11T22:23:35.4205204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4205649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4206216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4206664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4207241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4207684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4208249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4208691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4209221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4209712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4210350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4211027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4211603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4212078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4212411Z dist init r=1, world=2
2023-01-11T22:23:35.4212658Z dist init r=0, world=2
2023-01-11T22:23:35.4212895Z ok (4.512s)
2023-01-11T22:23:35.4213384Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60047
2023-01-11T22:23:35.4213979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60048
2023-01-11T22:23:35.4214585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4215029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4215586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4216045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4216614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4217048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4217594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4218051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4218496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4218967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4219617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4220297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4220872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4221326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4221673Z dist init r=1, world=2
2023-01-11T22:23:35.4221923Z dist init r=0, world=2
2023-01-11T22:23:35.4222143Z ok (4.512s)
2023-01-11T22:23:35.4222655Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60130
2023-01-11T22:23:35.4223251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60131
2023-01-11T22:23:35.4223858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4224285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4224850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4225308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4225873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4226361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4226924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4227377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4227806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4228346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4229003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4229678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4230180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4230641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4230989Z dist init r=0, world=2
2023-01-11T22:23:35.4231221Z dist init r=1, world=2
2023-01-11T22:23:35.4231456Z ok (4.412s)
2023-01-11T22:23:35.4231850Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4232558Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60213
2023-01-11T22:23:35.4233107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60214
2023-01-11T22:23:35.4233707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4234151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4234714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4235174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4235751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4236190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4236738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4237197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4237645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4238140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4238775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4239675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4240204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4240647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4240998Z dist init r=1, world=2
2023-01-11T22:23:35.4241254Z dist init r=0, world=2
2023-01-11T22:23:35.4241489Z ok (4.011s)
2023-01-11T22:23:35.4241864Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4242572Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60292
2023-01-11T22:23:35.4243100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60293
2023-01-11T22:23:35.4243775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4244219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4244791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4245254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4245875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4246325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4246896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4247335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4247786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4248279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4248928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4249593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4250110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4250574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4250927Z dist init r=1, world=2
2023-01-11T22:23:35.4251158Z dist init r=0, world=2
2023-01-11T22:23:35.4251397Z ok (3.910s)
2023-01-11T22:23:35.4251799Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4252502Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60371
2023-01-11T22:23:35.4253028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60372
2023-01-11T22:23:35.4253628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4254075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4254637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4255099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4255669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4256096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4256659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4257117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4257566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4258042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4258695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4259377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4259885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4260395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4260741Z dist init r=0, world=2
2023-01-11T22:23:35.4260988Z dist init r=1, world=2
2023-01-11T22:23:35.4261208Z ok (3.912s)
2023-01-11T22:23:35.4261610Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4262327Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60450
2023-01-11T22:23:35.4262880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60451
2023-01-11T22:23:35.4263491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4263936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4264503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4264953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4265528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4265964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4266528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4266973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4267418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4267910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4268541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4269229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4269743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4270205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4270539Z dist init r=0, world=2
2023-01-11T22:23:35.4270787Z dist init r=1, world=2
2023-01-11T22:23:35.4271023Z ok (3.911s)
2023-01-11T22:23:35.4271407Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4272110Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60529
2023-01-11T22:23:35.4272634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60530
2023-01-11T22:23:35.4273229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4273658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4274213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4274653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4275204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4275666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4276250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4276709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4277141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4277695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4278347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4279027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4279726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4280269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4280627Z dist init r=0, world=2
2023-01-11T22:23:35.4280859Z dist init r=1, world=2
2023-01-11T22:23:35.4281095Z ok (4.011s)
2023-01-11T22:23:35.4281501Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:23:35.4282226Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60608
2023-01-11T22:23:35.4282731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60609
2023-01-11T22:23:35.4283325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4283771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4284325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4284788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4285358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:23:35.4285799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:35.4286346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:35.4286809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:35.4287255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:35.4287733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:35.4288388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4289066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:35.4289580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:35.4290027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:35.4290373Z dist init r=1, world=2
2023-01-11T22:23:35.4290621Z dist init r=0, world=2
2023-01-11T22:23:35.4290841Z ok (4.011s)
2023-01-11T22:23:35.4290992Z 
2023-01-11T22:23:35.4291261Z ----------------------------------------------------------------------
2023-01-11T22:23:35.4291596Z Ran 27 tests in 118.843s
2023-01-11T22:23:35.4291759Z 
2023-01-11T22:23:35.4291852Z OK
2023-01-11T22:23:35.4291966Z 
2023-01-11T22:23:35.4292089Z Generating XML reports...
2023-01-11T22:23:35.4292710Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20230111222136.xml
2023-01-11T22:23:35.4293078Z 
2023-01-11T22:23:35.4293572Z ##[endgroup]
2023-01-11T22:23:35.4294171Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_4f9sjntz)
2023-01-11T22:23:35.4294522Z 
2023-01-11T22:23:35.4294827Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2023-01-11 22:23:35.391610]
2023-01-11T22:23:35.4295652Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:35.391919]
2023-01-11T22:26:06.8854998Z 
2023-01-11T22:26:06.8857881Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T22:26:06.8859169Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_9j8rl2gu)
2023-01-11T22:26:06.8859598Z 
2023-01-11T22:26:06.8859711Z Running tests...
2023-01-11T22:26:06.8860233Z ----------------------------------------------------------------------
2023-01-11T22:26:06.8862255Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2023-01-11T22:26:06.8862817Z   test_empty (__main__.TestCreateTensorFromParams) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:06.8863202Z ok (1.620s)
2023-01-11T22:26:06.8863610Z   test_local_tensor (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60722
2023-01-11T22:26:06.8864103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60723
2023-01-11T22:26:06.8864559Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60724
2023-01-11T22:26:06.8866867Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60725
2023-01-11T22:26:06.8867579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8868052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8868631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8869118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8869683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8870127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8870701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8872232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8872848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8873308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8873868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8874287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8874868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8875330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8875914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8876358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8876798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8877284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8877752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8878192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8878575Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.8879234Z   test_local_tensor_error (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60858
2023-01-11T22:26:06.8880104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60859
2023-01-11T22:26:06.8880551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60860
2023-01-11T22:26:06.8880986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60861
2023-01-11T22:26:06.8881702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8882143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8882736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8883198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8883776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8884199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8884770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8885230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8885790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8886233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8886801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8887264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8887879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8888298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8888862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8889327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8889760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8890217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8890684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8891139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8891500Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.8891971Z   test_collect_local_shard (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60994
2023-01-11T22:26:06.8892497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60995
2023-01-11T22:26:06.8892941Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60996
2023-01-11T22:26:06.8893359Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60997
2023-01-11T22:26:06.8893975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8894429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8894982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8895840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8896444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8897005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8897571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8898046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8898623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8899114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8899684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8900149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8900724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8901169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8901727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8902190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8902629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8903086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8903549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8904024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8904410Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.8904857Z   test_reshard_output (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61130
2023-01-11T22:26:06.8905381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61131
2023-01-11T22:26:06.8905827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61132
2023-01-11T22:26:06.8906405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61133
2023-01-11T22:26:06.8907315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8907771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8908353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8908805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8909393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8909843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8910418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8910865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8911442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8911880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8912451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8912899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8913466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8913902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8914547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8915007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8915439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8915906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8916401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8916869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8917255Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.8917723Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61266
2023-01-11T22:26:06.8918262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61267
2023-01-11T22:26:06.8918705Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61268
2023-01-11T22:26:06.8919141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61269
2023-01-11T22:26:06.8920072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8920518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8921089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8921552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8922113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8922552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8923123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8923562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8924131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8924567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8925130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8925570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8926140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8926574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8927123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8927582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8928013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8929045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8929493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8929947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8930333Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:26:06.8930799Z   test_shard_metadata_init (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61402
2023-01-11T22:26:06.8931296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61403
2023-01-11T22:26:06.8931842Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61404
2023-01-11T22:26:06.8932280Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61405
2023-01-11T22:26:06.8932872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8933315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8933943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8934412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8934972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8935410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8935974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8936417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8936992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8937428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8937993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8938439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8939011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8939449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8940014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8940454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8940885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8941350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8941794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8942250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8942638Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.8943104Z   test_shard_parameter (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61538
2023-01-11T22:26:06.8943605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61539
2023-01-11T22:26:06.8944050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61540
2023-01-11T22:26:06.8944489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61541
2023-01-11T22:26:06.8945075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8945519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8946134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8946601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8947158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8947592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8948156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8948738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8949298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8949734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8950298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8950786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8951364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8951796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8952356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8952800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8953230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8953693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8954138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8954599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8954983Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.8955462Z   test_shard_parameter_errors (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61674
2023-01-11T22:26:06.8955970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61675
2023-01-11T22:26:06.8956409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61676
2023-01-11T22:26:06.8956849Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61677
2023-01-11T22:26:06.8957438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8957884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8958451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8958911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8959800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8960252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8960824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8961279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8961833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8962269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8962830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8963269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8963839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8964272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8964835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8965275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8965816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8966284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8966732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8967192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8967573Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.8968083Z   test_shard_tensor (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61810
2023-01-11T22:26:06.8968578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61811
2023-01-11T22:26:06.8969015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61812
2023-01-11T22:26:06.8969448Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61813
2023-01-11T22:26:06.8970061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8970487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8971054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8971521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8972080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8972522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8973084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8973541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8974097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8974538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8975101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8975539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8976110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8976547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8977110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8977547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8977978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8978454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8978896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8979354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8979733Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.8980197Z   test_shard_tensor_errors (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61946
2023-01-11T22:26:06.8980688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61947
2023-01-11T22:26:06.8981125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61948
2023-01-11T22:26:06.8981559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61949
2023-01-11T22:26:06.8982161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8982667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8983238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8983695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8984248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8984731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8985308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8985769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8986323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8986765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8987330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8987766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8988335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8988779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8989344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8989779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8990213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.8990682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.8991142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.8991582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.8991960Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.8992429Z   test_cleanup (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62082
2023-01-11T22:26:06.8992936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62083
2023-01-11T22:26:06.8993375Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62084
2023-01-11T22:26:06.8993815Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62085
2023-01-11T22:26:06.8994419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8994847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8995412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8995873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8996429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8996868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8997430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.8997886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.8998439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.8998943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.8999867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9000329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9000891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9001332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9001966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9002421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9002853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9003320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9003787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9004232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9004610Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9005095Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62218
2023-01-11T22:26:06.9005617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62219
2023-01-11T22:26:06.9006058Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62220
2023-01-11T22:26:06.9006490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62221
2023-01-11T22:26:06.9007096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9007527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9008094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9008556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9009109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9009548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9010117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9010577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9011129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9011567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9012135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9012593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9013140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9013578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9014148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9014588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9015019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9015485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9016050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9016494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9016872Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9017246Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2023-01-11T22:26:06.9017766Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62354
2023-01-11T22:26:06.9018343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62355
2023-01-11T22:26:06.9018788Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62356
2023-01-11T22:26:06.9019225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62357
2023-01-11T22:26:06.9019814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9020260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9020824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9021288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9021843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9022284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9022845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9023289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9023860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9024297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9024863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9025304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9025872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9026306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9026857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9027314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9027745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9028211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9028655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9029106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9029494Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9029856Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2023-01-11T22:26:06.9030344Z Test sharded_tensor.full(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62490
2023-01-11T22:26:06.9030834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62491
2023-01-11T22:26:06.9031274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62492
2023-01-11T22:26:06.9031692Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62493
2023-01-11T22:26:06.9032289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9032793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9033345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9033811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9034384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9034867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9035427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9035886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9036450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9036891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9037439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9037895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9038464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9038881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9039775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9040240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9040669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9041119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9041582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9042042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9042411Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9042787Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2023-01-11T22:26:06.9043274Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62626
2023-01-11T22:26:06.9043756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62627
2023-01-11T22:26:06.9044182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62628
2023-01-11T22:26:06.9044612Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62629
2023-01-11T22:26:06.9045221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9045668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9046256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9046702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9047268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9047715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9048294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9048751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9049320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9049826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9050395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9050853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9051407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9051843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9052464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9052930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9053345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9053811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9054276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9054720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9055107Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9055487Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2023-01-11T22:26:06.9055985Z Test sharded_tensor.rand(...)/randn(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62762
2023-01-11T22:26:06.9056462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62763
2023-01-11T22:26:06.9056899Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62764
2023-01-11T22:26:06.9057335Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62765
2023-01-11T22:26:06.9057942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9058374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9058932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9059369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9059919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9060382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9060958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9061419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9061969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9062413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9062979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9063426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9063997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9064435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9065001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9065438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9065868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9066333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9066838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9067297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9067676Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9068057Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2023-01-11T22:26:06.9068573Z Test sharded_tensor.zeros(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62898
2023-01-11T22:26:06.9069064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62899
2023-01-11T22:26:06.9069502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62900
2023-01-11T22:26:06.9069919Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62901
2023-01-11T22:26:06.9070524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9070972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9071541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9071988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9072563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9073006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9073568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9074008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9074574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9075013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9075560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9076018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9076589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9077028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9077576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9078029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9078456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9078905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9079364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9080156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9080542Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9080874Z   test_gather_even (__main__.TestShardedTensorChunked)
2023-01-11T22:26:06.9081387Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63034
2023-01-11T22:26:06.9081914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63035
2023-01-11T22:26:06.9082354Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63036
2023-01-11T22:26:06.9082775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63037
2023-01-11T22:26:06.9083468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9083911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9084463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9084925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9085495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9085995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9086556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9087007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9087575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9087998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9088558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9089014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9089580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9090000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9090565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9091018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9091449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9091902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9092354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9092810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9093179Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9093532Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2023-01-11T22:26:06.9094045Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63170
2023-01-11T22:26:06.9094575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63171
2023-01-11T22:26:06.9095001Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63172
2023-01-11T22:26:06.9095428Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63173
2023-01-11T22:26:06.9096033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9096462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9097031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9097495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9098067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9098486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9099052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9099509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9100060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9100570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9101136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9101590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9102139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9102634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9103209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9103668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9104082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9104554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9105014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9105458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9105838Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9106337Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63306
2023-01-11T22:26:06.9106881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63307
2023-01-11T22:26:06.9107303Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63308
2023-01-11T22:26:06.9107731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63309
2023-01-11T22:26:06.9108333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9108761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9109331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9109793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9110363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9110788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9111353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9111810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9112380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9112800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9113365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9113819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9114371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9114812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9115373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9115827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9116239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9116767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9117228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9117673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9118061Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9118547Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63442
2023-01-11T22:26:06.9119131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63443
2023-01-11T22:26:06.9119932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63444
2023-01-11T22:26:06.9120369Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63445
2023-01-11T22:26:06.9120981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9121414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9121981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9122450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9123020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9123445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9123999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9124434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9124999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9125449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9126029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9126486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9127044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9127482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9128050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9128505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9128918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9129379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9129851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9130292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9130680Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9131164Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63578
2023-01-11T22:26:06.9131698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63579
2023-01-11T22:26:06.9132122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63580
2023-01-11T22:26:06.9132551Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63581
2023-01-11T22:26:06.9133153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9133678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9134236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9134699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9135268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9135689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9136318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9136786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9137359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9137774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9138343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9138797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9139349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9139778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9140344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9140795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9141206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9141673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9142138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9142600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9142967Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9143454Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63714
2023-01-11T22:26:06.9143990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63715
2023-01-11T22:26:06.9144418Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63716
2023-01-11T22:26:06.9144854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63717
2023-01-11T22:26:06.9145458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9145942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9146501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9146962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9147535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9147958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9148521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9148980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9149548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9149964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9150526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9151043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9151596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9152027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9152589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9153086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9153508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9153970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9154430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9154893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9155262Z skip: Need at least 4 CUDA devices (2.510s)
2023-01-11T22:26:06.9155753Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63850
2023-01-11T22:26:06.9156292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63851
2023-01-11T22:26:06.9156719Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63852
2023-01-11T22:26:06.9157147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63853
2023-01-11T22:26:06.9157749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9158188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9158736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9159198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9159942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9160367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9160931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9161392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9161964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9162379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9162943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9163401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9163975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9164390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9164950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9165411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9165877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9166345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9166805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9167351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9167720Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9168193Z   test_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63986
2023-01-11T22:26:06.9168715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63987
2023-01-11T22:26:06.9169139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63988
2023-01-11T22:26:06.9169631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63989
2023-01-11T22:26:06.9170243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9170684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9171234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9171695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9172268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9172704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9173257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9173718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9174289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9174707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9175275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9175735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9176301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9176716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9177274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9177727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9178141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9178603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9179058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9179516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9179887Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9180372Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64122
2023-01-11T22:26:06.9180902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64123
2023-01-11T22:26:06.9181327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64124
2023-01-11T22:26:06.9181763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64125
2023-01-11T22:26:06.9182367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9182810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9183360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9183884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9184452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9184890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9185437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9185891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9186501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9186930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9187492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9187946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9188518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9188931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9189497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9189948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9190368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9190836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9191290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9191751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9192117Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9192610Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64258
2023-01-11T22:26:06.9193146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64259
2023-01-11T22:26:06.9193588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64260
2023-01-11T22:26:06.9194008Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64261
2023-01-11T22:26:06.9194608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9195054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9195604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9196065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9196646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9197082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9197615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9198054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9198628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9199072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9199823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9200286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9200858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9201364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9201932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9202392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9202822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9203332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9203797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9204260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9204631Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9205125Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64394
2023-01-11T22:26:06.9205660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64395
2023-01-11T22:26:06.9206100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64396
2023-01-11T22:26:06.9206522Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64397
2023-01-11T22:26:06.9207127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9207570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9208117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9208580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9209164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9209608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9210155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9210614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9211180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9211600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9212161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9212617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9213182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9213602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9214165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9214620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9215050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9215502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9215957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9216417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9216786Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9217271Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64530
2023-01-11T22:26:06.9217876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64531
2023-01-11T22:26:06.9218313Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64532
2023-01-11T22:26:06.9218732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64533
2023-01-11T22:26:06.9219330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9219841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9220409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9220871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9221439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9221879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9222429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9222883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9223451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9223886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9224438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9224897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9225459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9225879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9226442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9226894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9227323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9227773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9228227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9228688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9229056Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9229530Z   test_state_dict (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64666
2023-01-11T22:26:06.9230055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64667
2023-01-11T22:26:06.9230494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64668
2023-01-11T22:26:06.9230916Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64669
2023-01-11T22:26:06.9231512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9231952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9232504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9232963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9233533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9233971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9234584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9235040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9235609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9236043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9236633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9237098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9237663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9238081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9238647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9239099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9239682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9240139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9240597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9241057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9241425Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9241912Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64802
2023-01-11T22:26:06.9242442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64803
2023-01-11T22:26:06.9242886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64804
2023-01-11T22:26:06.9243305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64805
2023-01-11T22:26:06.9243906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9244345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9244914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9245361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9245972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9246411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9246954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9247391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9247956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9248413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9248979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9249437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9250008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9250428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9250997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9251541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9251973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9252420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9252873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9253386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9253784Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9254267Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64938
2023-01-11T22:26:06.9254812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64939
2023-01-11T22:26:06.9255257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64940
2023-01-11T22:26:06.9255674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64941
2023-01-11T22:26:06.9256273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9256714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9257282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9257730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9258299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9258735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9259285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9259747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9260315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9260749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9261298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9261755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9262324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9262758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9263305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9263764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9264193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9264643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9265096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9265561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9265947Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9266409Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65074
2023-01-11T22:26:06.9266933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65075
2023-01-11T22:26:06.9267374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65076
2023-01-11T22:26:06.9267854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65077
2023-01-11T22:26:06.9268453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9268893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9269462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9269970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9270554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9270990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9271538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9272005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9272573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9273006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9273554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9274015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9274584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9275015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9275562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9276023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9276449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9276895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9277358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9277813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9278201Z skip: Need at least 4 CUDA devices (2.413s)
2023-01-11T22:26:06.9278673Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65210
2023-01-11T22:26:06.9279207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65211
2023-01-11T22:26:06.9279817Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65212
2023-01-11T22:26:06.9280243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65213
2023-01-11T22:26:06.9280854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9281296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9281857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9282280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9282851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9283312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9283891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9284418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9284995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9285431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9285981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9286437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9287060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9287507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9288059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9288513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9288945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9289394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9289856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9290307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9290688Z skip: Need at least 4 CUDA devices (2.510s)
2023-01-11T22:26:06.9291169Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65346
2023-01-11T22:26:06.9291704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65347
2023-01-11T22:26:06.9292139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65348
2023-01-11T22:26:06.9292559Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65349
2023-01-11T22:26:06.9293168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9293610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9294178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9294623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9295197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9295633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9296199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9296644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9297214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9297650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9298197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9298653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9299222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9299653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9300199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9300655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9301082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9301594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9302057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9302513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9302902Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9303321Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2023-01-11T22:26:06.9303823Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65482
2023-01-11T22:26:06.9304306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65483
2023-01-11T22:26:06.9304749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65484
2023-01-11T22:26:06.9305171Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65485
2023-01-11T22:26:06.9305776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9306216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9306770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9307228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9307800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9308232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9308780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9309234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9309807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9310223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9310787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9311238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9311810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9312230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9312796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9313250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9313666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9314132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9335270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9335832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9336236Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9336592Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2023-01-11T22:26:06.9337126Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65618
2023-01-11T22:26:06.9337666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65619
2023-01-11T22:26:06.9338120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65620
2023-01-11T22:26:06.9338727Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65621
2023-01-11T22:26:06.9339355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9339783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9340333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9340784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9341408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9341841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9342393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9342850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9343426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9343848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9344418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9344874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9345453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9345924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9346493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9346947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9347388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9347842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9348297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9348763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9349132Z skip: Need at least 4 CUDA devices (2.510s)
2023-01-11T22:26:06.9349501Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2023-01-11T22:26:06.9350027Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65754
2023-01-11T22:26:06.9350560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65755
2023-01-11T22:26:06.9350989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65756
2023-01-11T22:26:06.9351434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65757
2023-01-11T22:26:06.9352042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9352472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9353047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9353511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9354088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9354508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9355071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9355597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9356154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9356591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9357154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9357613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9358210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9358662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9359230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9359955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9360382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9360852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9361318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9361765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9362148Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9362640Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65890
2023-01-11T22:26:06.9363182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65891
2023-01-11T22:26:06.9363610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65892
2023-01-11T22:26:06.9364045Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65893
2023-01-11T22:26:06.9364663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9365095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9365665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9366126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9366702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9367118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9367685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9368141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9368712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9369130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9369695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9370153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9370712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9371157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9371722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9372177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9372684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9373151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9373611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9374060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9374450Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9375009Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66026
2023-01-11T22:26:06.9375563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66027
2023-01-11T22:26:06.9375996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66028
2023-01-11T22:26:06.9376438Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66029
2023-01-11T22:26:06.9377051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9377478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9378048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9378517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9379092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9379514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9380080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9380534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9381109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9381528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9382093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9382551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9383106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9383544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9384110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9384564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9384981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9385454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9385919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9386364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9386749Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9387235Z   test_new_group (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66162
2023-01-11T22:26:06.9387766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66163
2023-01-11T22:26:06.9388194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66164
2023-01-11T22:26:06.9388635Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66165
2023-01-11T22:26:06.9389241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9389743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9390294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9390757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9391124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9391343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9391727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9391915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9392258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9392436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9392806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9392994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9393359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9393535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9393906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9394093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9394307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9394540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9394760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9394983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9395133Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9395458Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66298
2023-01-11T22:26:06.9395679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66299
2023-01-11T22:26:06.9395891Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66300
2023-01-11T22:26:06.9396089Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66301
2023-01-11T22:26:06.9396463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9396642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9397023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9397211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9397797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9398127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9398519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9398709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9399054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9399304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9399941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9400130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9400498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9400672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9401121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9401317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9401527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9401754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9401979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9402206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9402356Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9402687Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66434
2023-01-11T22:26:06.9402911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66435
2023-01-11T22:26:06.9403125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66436
2023-01-11T22:26:06.9403337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66437
2023-01-11T22:26:06.9403695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9403873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9404252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9404443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9404804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9404976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9405354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9405542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9405885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9406057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9406430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9406617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9406989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9407156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9407523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9407710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9407940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9408150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9408479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9408909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9409189Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9409831Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66570
2023-01-11T22:26:06.9410076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66571
2023-01-11T22:26:06.9410361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66572
2023-01-11T22:26:06.9410583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66573
2023-01-11T22:26:06.9410953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9411129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9411517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9411705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9412073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9412248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9412623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9412814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9413180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9413335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9413712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9413901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9414271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9414442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9414816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9415007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9415234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9415445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9415667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9415893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9416044Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9416374Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66706
2023-01-11T22:26:06.9416592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66707
2023-01-11T22:26:06.9416810Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66708
2023-01-11T22:26:06.9417028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66709
2023-01-11T22:26:06.9417401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9417557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9417936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9418187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9418555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9418726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9419098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9419330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9419699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9419851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9420219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9420407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9420773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9420944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9421311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9421499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9421726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9421952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9422157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9422385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9422534Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9422860Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66842
2023-01-11T22:26:06.9423079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66843
2023-01-11T22:26:06.9423295Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66844
2023-01-11T22:26:06.9423512Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66845
2023-01-11T22:26:06.9423883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9424039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9424417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9424610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9424976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9425148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9425519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9425715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9426082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9426253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9426613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9426874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9427240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9427411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9427781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9427966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9428238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9428473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9428678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9428904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9429061Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9429392Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66978
2023-01-11T22:26:06.9429610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66979
2023-01-11T22:26:06.9429827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66980
2023-01-11T22:26:06.9430048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66981
2023-01-11T22:26:06.9430424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9430600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9430962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9431156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9431517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9431688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9432056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9432229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9432607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9432796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9433177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9433348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9433716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9433891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9434259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9434445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9434676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9434908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9435129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9435337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9435546Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9435866Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67114
2023-01-11T22:26:06.9436080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67115
2023-01-11T22:26:06.9436294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67116
2023-01-11T22:26:06.9436509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67117
2023-01-11T22:26:06.9436922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9437102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9437465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9437654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9438021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9438193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9438566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9438756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9439119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9439293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9439907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9440079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9440454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9440628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9440996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9441181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9441410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9441639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9441860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9442067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9442218Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9442544Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67250
2023-01-11T22:26:06.9442764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67251
2023-01-11T22:26:06.9442978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67252
2023-01-11T22:26:06.9443195Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67253
2023-01-11T22:26:06.9443571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9443747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9444125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9444297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9444787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9444957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9445330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9445519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9445980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9446161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9446534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9446702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9447068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9447244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9447611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9447796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9448025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9448256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9448478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9448698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9448829Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9449765Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78068 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2023-01-11T22:26:06.9450140Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67386
2023-01-11T22:26:06.9450359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67387
2023-01-11T22:26:06.9450574Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67388
2023-01-11T22:26:06.9450789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67389
2023-01-11T22:26:06.9451160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9451338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9451718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9451910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9452257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9452429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9452803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9452991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9453352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9453581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9453961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9454148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9454488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9454661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9455075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9455269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9455499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9455727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9455956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9456179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9456334Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:26:06.9456705Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67522
2023-01-11T22:26:06.9456926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67523
2023-01-11T22:26:06.9457139Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67524
2023-01-11T22:26:06.9457353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67525
2023-01-11T22:26:06.9457734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9457914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9458293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9458484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9458828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9459001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9459376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9459563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9459926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9460101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9460476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9460665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9461032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9461187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9461557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9461744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9461972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9462196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9462474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9462704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9462854Z skip: Need at least 4 CUDA devices (2.413s)
2023-01-11T22:26:06.9463204Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67658
2023-01-11T22:26:06.9463463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67659
2023-01-11T22:26:06.9463685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67660
2023-01-11T22:26:06.9463901Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67661
2023-01-11T22:26:06.9464278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9464457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9464834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9465024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9465390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9465543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9465982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9466173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9466536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9466708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9467080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9467264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9467633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9467802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9468157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9468345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9468572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9468799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9469023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9469248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9469400Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9469766Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67794
2023-01-11T22:26:06.9469971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67795
2023-01-11T22:26:06.9470187Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67796
2023-01-11T22:26:06.9470400Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67797
2023-01-11T22:26:06.9470772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9470947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9471381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9471572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9471935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9472107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9472503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9472699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9473059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9473228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9473601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9473788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9474155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9474325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9474678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9474865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9475092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9475317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9475540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9475769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9476016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:06.9476258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:26:06.9476502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:06.9476723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:26:06.9477129Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:26:06.9477529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:26:06.9477925Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:26:06.9478320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:26:06.9478469Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9478847Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67942
2023-01-11T22:26:06.9479068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67943
2023-01-11T22:26:06.9479284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67944
2023-01-11T22:26:06.9479696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67945
2023-01-11T22:26:06.9480084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9480340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9480721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9480911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9481274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9481497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9481882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9482073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9482424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9482602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9482971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9483156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9483514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9483689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9484061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9484246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9484456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9484682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9484914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9485134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9485282Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9485651Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68078
2023-01-11T22:26:06.9485872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68079
2023-01-11T22:26:06.9486088Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68080
2023-01-11T22:26:06.9486302Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68081
2023-01-11T22:26:06.9486658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9486836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9487218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9487405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9487768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9487942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9488317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9488506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9488857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9489032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9489480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9489667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9490030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9490203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9490613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9490805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9491039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9491250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9491482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9491703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9491852Z skip: Need at least 4 CUDA devices (2.510s)
2023-01-11T22:26:06.9492226Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68214
2023-01-11T22:26:06.9492446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68215
2023-01-11T22:26:06.9492661Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68216
2023-01-11T22:26:06.9492874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68217
2023-01-11T22:26:06.9493234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9493410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9493788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9493982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9494344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9494520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9494898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9495085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9495452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9495606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9495981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9496169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9496528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9496699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9497071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9497256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9497481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9497691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9497975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9498194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9498342Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9498696Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68350
2023-01-11T22:26:06.9498915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68351
2023-01-11T22:26:06.9499171Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68352
2023-01-11T22:26:06.9499392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68353
2023-01-11T22:26:06.9499765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9499921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9500301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9500487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9500847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9501019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9501395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9501582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9501948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9502101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9502476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9502659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9503019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9503193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9503565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9503752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9503982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9504212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9504420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9504644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9504794Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9505128Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68486
2023-01-11T22:26:06.9505346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68487
2023-01-11T22:26:06.9505565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68488
2023-01-11T22:26:06.9505780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68489
2023-01-11T22:26:06.9506152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9506308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9506744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9506932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9507294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9507466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9507888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9508081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9508452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9508624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9508978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9509168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9509530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9509702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9510069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9510259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9510489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9510717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9510927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9511152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9511298Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9511676Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68622
2023-01-11T22:26:06.9511892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68623
2023-01-11T22:26:06.9512108Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68624
2023-01-11T22:26:06.9512322Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68625
2023-01-11T22:26:06.9512696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9512870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9513227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9513421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9513786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9513952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9514329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9514524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9514883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9515054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9515408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9515664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9516024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9516194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9516561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9516790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9517024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9517253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9517478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9517684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9517835Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:26:06.9518177Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68758
2023-01-11T22:26:06.9518394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68759
2023-01-11T22:26:06.9518610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68760
2023-01-11T22:26:06.9518827Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68761
2023-01-11T22:26:06.9519201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9519593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9519971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9520168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9520526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9520698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9521069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9521260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9521627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9521798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9522167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9522339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9522702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9522876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9523248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9523432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9523665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9523891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9524117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9524323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9524555Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9524907Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68894
2023-01-11T22:26:06.9525126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68895
2023-01-11T22:26:06.9525338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68896
2023-01-11T22:26:06.9525604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68897
2023-01-11T22:26:06.9525987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9526162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9526542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9526720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9527080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9527250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9527621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9527808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9528179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9528352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9528727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9528915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9529262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:26:06.9529436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:06.9529812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:06.9530000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:06.9530257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:06.9530695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:26:06.9531015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:06.9531239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:26:06.9531379Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:26:06.9531605Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.053s)
2023-01-11T22:26:06.9531630Z 
2023-01-11T22:26:06.9531907Z ----------------------------------------------------------------------
2023-01-11T22:26:06.9532025Z Ran 64 tests in 149.134s
2023-01-11T22:26:06.9532045Z 
2023-01-11T22:26:06.9532154Z OK (skipped=62)
2023-01-11T22:26:06.9532173Z 
2023-01-11T22:26:06.9532297Z Generating XML reports...
2023-01-11T22:26:06.9532814Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20230111222337.xml
2023-01-11T22:26:06.9533317Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20230111222337.xml
2023-01-11T22:26:06.9533773Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20230111222337.xml
2023-01-11T22:26:06.9534297Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20230111222337.xml
2023-01-11T22:26:06.9534749Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20230111222337.xml
2023-01-11T22:26:06.9535218Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20230111222337.xml
2023-01-11T22:26:06.9535715Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20230111222337.xml
2023-01-11T22:26:06.9536216Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20230111222337.xml
2023-01-11T22:26:06.9536712Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20230111222337.xml
2023-01-11T22:26:06.9537220Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20230111222337.xml
2023-01-11T22:26:06.9537741Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20230111222337.xml
2023-01-11T22:26:06.9538269Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20230111222337.xml
2023-01-11T22:26:06.9538291Z 
2023-01-11T22:26:06.9538810Z ##[endgroup]
2023-01-11T22:26:06.9539364Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_9j8rl2gu)
2023-01-11T22:26:06.9539385Z 
2023-01-11T22:26:06.9539666Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2023-01-11 22:26:06.888027]
2023-01-11T22:26:06.9540164Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:26:06.888345]
2023-01-11T22:29:44.9946018Z 
2023-01-11T22:29:44.9946527Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params
2023-01-11T22:29:44.9950296Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_kixem_8r)
2023-01-11T22:29:44.9950733Z 
2023-01-11T22:29:44.9950878Z Running tests...
2023-01-11T22:29:44.9951711Z ----------------------------------------------------------------------
2023-01-11T22:29:44.9952323Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2023-01-11T22:29:44.9953163Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:44.9953670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69065
2023-01-11T22:29:44.9954418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69066
2023-01-11T22:29:44.9957475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9957947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:44.9958569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:44.9959186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:44.9960407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9960897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:44.9961776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:44.9962253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:44.9962697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:44.9963199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:44.9964023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:44.9964745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:44.9965276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:44.9965751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:44.9967034Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:44.9967873Z   warnings.warn(
2023-01-11T22:29:44.9969356Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:44.9970409Z   warnings.warn(
2023-01-11T22:29:44.9970650Z dist init r=0, world=2
2023-01-11T22:29:44.9970904Z dist init r=1, world=2
2023-01-11T22:29:44.9971148Z ok (6.068s)
2023-01-11T22:29:44.9971588Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69148
2023-01-11T22:29:44.9972138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69149
2023-01-11T22:29:44.9972887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9973423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:44.9973989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:44.9974460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:44.9975048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9975503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:44.9976059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:44.9976524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:44.9977018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:44.9977500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:44.9978161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:44.9978844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:44.9979470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:44.9979922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:44.9981242Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:44.9982023Z   warnings.warn(
2023-01-11T22:29:44.9983192Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:44.9983970Z   warnings.warn(
2023-01-11T22:29:44.9984201Z dist init r=1, world=2
2023-01-11T22:29:44.9984453Z dist init r=0, world=2
2023-01-11T22:29:44.9984770Z ok (4.011s)
2023-01-11T22:29:44.9985130Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams)
2023-01-11T22:29:44.9985685Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69227
2023-01-11T22:29:44.9986188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69228
2023-01-11T22:29:44.9987033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9987489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:44.9988064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:44.9988521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:44.9989099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9989545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:44.9990125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:44.9990578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:44.9991032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:44.9991527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:44.9992161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:44.9992844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:44.9993369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:44.9993848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:44.9994181Z dist init r=0, world=2
2023-01-11T22:29:44.9994431Z dist init r=1, world=2
2023-01-11T22:29:44.9994672Z ok (4.012s)
2023-01-11T22:29:44.9995008Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams)
2023-01-11T22:29:44.9995557Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69306
2023-01-11T22:29:44.9996166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69307
2023-01-11T22:29:44.9996787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9997218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:44.9997798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:44.9998317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:44.9998916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:44.9999346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0000741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0001222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0001658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0002161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0002825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0003522Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0004024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0004500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0004861Z dist init r=0, world=2
2023-01-11T22:29:45.0005120Z dist init r=1, world=2
2023-01-11T22:29:45.0005347Z ok (3.911s)
2023-01-11T22:29:45.0005721Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams)
2023-01-11T22:29:45.0006282Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69385
2023-01-11T22:29:45.0006786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69386
2023-01-11T22:29:45.0007402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0007860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0008436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0008887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0009468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0009918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0010476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0010947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0011402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0011902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0012535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0013229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0013747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0014345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0014681Z dist init r=1, world=2
2023-01-11T22:29:45.0014932Z dist init r=0, world=2
2023-01-11T22:29:45.0015174Z ok (4.011s)
2023-01-11T22:29:45.0015524Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams)
2023-01-11T22:29:45.0016080Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69464
2023-01-11T22:29:45.0016662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69465
2023-01-11T22:29:45.0017286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0017716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0018288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0018759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0019317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0019757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0020327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0020787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0021221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0021713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0022368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0023061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0023561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0024032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0024384Z dist init r=1, world=2
2023-01-11T22:29:45.0024616Z dist init r=0, world=2
2023-01-11T22:29:45.0024852Z ok (4.011s)
2023-01-11T22:29:45.0025372Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69543
2023-01-11T22:29:45.0025973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69544
2023-01-11T22:29:45.0026565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0027015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0027583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0028029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0028604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0029047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0029615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0030055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0030503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0031062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0031699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0032385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0032905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0033419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0033762Z dist init r=1, world=2
2023-01-11T22:29:45.0034013Z dist init r=0, world=2
2023-01-11T22:29:45.0034257Z ok (3.911s)
2023-01-11T22:29:45.0034754Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69622
2023-01-11T22:29:45.0035361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69623
2023-01-11T22:29:45.0035971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0036417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0036971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0037440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0038014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0038450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0039001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0039844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0040306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0040781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0041450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0042142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0042666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0043116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0043466Z dist init r=1, world=2
2023-01-11T22:29:45.0043718Z dist init r=0, world=2
2023-01-11T22:29:45.0043942Z ok (4.012s)
2023-01-11T22:29:45.0044457Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69701
2023-01-11T22:29:45.0045053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69702
2023-01-11T22:29:45.0045660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0046089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0046669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0047137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0047713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0048227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0048801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0049267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0049701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0050195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0050905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0051602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0052100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0052576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0053663Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0054337Z   warnings.warn(
2023-01-11T22:29:45.0055289Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0055943Z   warnings.warn(
2023-01-11T22:29:45.0056910Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0057564Z   warnings.warn(
2023-01-11T22:29:45.0058517Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0059138Z   warnings.warn(
2023-01-11T22:29:45.0059383Z dist init r=1, world=2
2023-01-11T22:29:45.0059631Z dist init r=0, world=2
2023-01-11T22:29:45.0059855Z ok (3.911s)
2023-01-11T22:29:45.0060371Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69780
2023-01-11T22:29:45.0060974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69781
2023-01-11T22:29:45.0061584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0062012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0062593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0063594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0064240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0064667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0065330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0065793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0066245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0066729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0067435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0068134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0068632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0069104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0070184Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0070845Z   warnings.warn(
2023-01-11T22:29:45.0071808Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0072433Z   warnings.warn(
2023-01-11T22:29:45.0073400Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0074066Z   warnings.warn(
2023-01-11T22:29:45.0075017Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0075663Z   warnings.warn(
2023-01-11T22:29:45.0075892Z dist init r=1, world=2
2023-01-11T22:29:45.0076141Z dist init r=0, world=2
2023-01-11T22:29:45.0076375Z ok (3.911s)
2023-01-11T22:29:45.0076873Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69859
2023-01-11T22:29:45.0077478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69860
2023-01-11T22:29:45.0078089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0078521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0079106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0079924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0080512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0080936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0081510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0082064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0082516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0082992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0083654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0084404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0084922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0085393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0085740Z dist init r=0, world=2
2023-01-11T22:29:45.0085994Z dist init r=1, world=2
2023-01-11T22:29:45.0086215Z ok (3.911s)
2023-01-11T22:29:45.0086727Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69938
2023-01-11T22:29:45.0087325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69939
2023-01-11T22:29:45.0087921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0088380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0088959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0089422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0089978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0090428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0090994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0091456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0091889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0092385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0093041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0093706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0094227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0094697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0095048Z dist init r=0, world=2
2023-01-11T22:29:45.0095284Z dist init r=1, world=2
2023-01-11T22:29:45.0095521Z ok (4.011s)
2023-01-11T22:29:45.0096034Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70017
2023-01-11T22:29:45.0096619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70018
2023-01-11T22:29:45.0097228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0097676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0098248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0098766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0099346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0099788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0100343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0100803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0101306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0101806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0102446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0103136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0103650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0104115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0104451Z dist init r=0, world=2
2023-01-11T22:29:45.0104702Z dist init r=1, world=2
2023-01-11T22:29:45.0104941Z ok (4.011s)
2023-01-11T22:29:45.0105438Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70096
2023-01-11T22:29:45.0106035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70097
2023-01-11T22:29:45.0106648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0107100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0107653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0108118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0108692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0109109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0109679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0110143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0110595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0111070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0111727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0112407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0112920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0113371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0113721Z dist init r=0, world=2
2023-01-11T22:29:45.0113973Z dist init r=1, world=2
2023-01-11T22:29:45.0114195Z ok (3.912s)
2023-01-11T22:29:45.0114708Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70175
2023-01-11T22:29:45.0115306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70176
2023-01-11T22:29:45.0115983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0116414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0116982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0117451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0118052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0118503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0119073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0119847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0120291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0120787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0121450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0122136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0122641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0123109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0123462Z dist init r=1, world=2
2023-01-11T22:29:45.0123693Z dist init r=0, world=2
2023-01-11T22:29:45.0123928Z ok (3.911s)
2023-01-11T22:29:45.0124437Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70254
2023-01-11T22:29:45.0125036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70255
2023-01-11T22:29:45.0125629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0126077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0126654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0127107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0127679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0128115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0128687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0129130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0129580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0130078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0130713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0131400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0131915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0132382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0132810Z dist init r=0, world=2
2023-01-11T22:29:45.0133063Z dist init r=1, world=2
2023-01-11T22:29:45.0133302Z ok (3.911s)
2023-01-11T22:29:45.0133797Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70333
2023-01-11T22:29:45.0134398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70334
2023-01-11T22:29:45.0135069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0135529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0136084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0136551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0137126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0137568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0138119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0138575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0139028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0139507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0140163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0140842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0141367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0141819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0142165Z dist init r=0, world=2
2023-01-11T22:29:45.0142418Z dist init r=1, world=2
2023-01-11T22:29:45.0142638Z ok (3.912s)
2023-01-11T22:29:45.0143148Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70412
2023-01-11T22:29:45.0143745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70413
2023-01-11T22:29:45.0144353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0144781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0145355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0145822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0146400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0146821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0147388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0147850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0148283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0148772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0149423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0150174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0150672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0151142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0151494Z dist init r=0, world=2
2023-01-11T22:29:45.0151726Z dist init r=1, world=2
2023-01-11T22:29:45.0151963Z ok (4.011s)
2023-01-11T22:29:45.0152527Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70491
2023-01-11T22:29:45.0153132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70492
2023-01-11T22:29:45.0153723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0154177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0154749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0155214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0155772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0156213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0156781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0157224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0157670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0158163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0158817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0159849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0160377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0160850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0161184Z dist init r=1, world=2
2023-01-11T22:29:45.0161440Z dist init r=0, world=2
2023-01-11T22:29:45.0161676Z ok (4.011s)
2023-01-11T22:29:45.0162185Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70570
2023-01-11T22:29:45.0162767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70571
2023-01-11T22:29:45.0163377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0163822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0164417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0164890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0165469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0165908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0166460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0167010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0167459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0167952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0168592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0169335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0169866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0170316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0170669Z dist init r=1, world=2
2023-01-11T22:29:45.0170921Z dist init r=0, world=2
2023-01-11T22:29:45.0171164Z ok (4.011s)
2023-01-11T22:29:45.0171659Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70649
2023-01-11T22:29:45.0172251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70650
2023-01-11T22:29:45.0172859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0173283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0173860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0174316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0174891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0175316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0175884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0176339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0176768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0177256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0177910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0178591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0179089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0179558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0179904Z dist init r=1, world=2
2023-01-11T22:29:45.0180154Z dist init r=0, world=2
2023-01-11T22:29:45.0180376Z ok (4.011s)
2023-01-11T22:29:45.0180881Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70728
2023-01-11T22:29:45.0181473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70729
2023-01-11T22:29:45.0182068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0182519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0183088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0183550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0184189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0184633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0185206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0185643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0186140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0186643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0187297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0187964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0188484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0188949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0189299Z dist init r=0, world=2
2023-01-11T22:29:45.0189529Z dist init r=1, world=2
2023-01-11T22:29:45.0189765Z ok (3.911s)
2023-01-11T22:29:45.0190084Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams)
2023-01-11T22:29:45.0190577Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70807
2023-01-11T22:29:45.0191092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70808
2023-01-11T22:29:45.0191696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0192141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0192703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0193164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0193738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0194161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0194732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0195187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0195637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0196107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0196760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0197438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0197939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0198410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0198767Z dist init r=0, world=2
2023-01-11T22:29:45.0199014Z dist init r=1, world=2
2023-01-11T22:29:45.0199234Z ok (3.910s)
2023-01-11T22:29:45.0200114Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70886
2023-01-11T22:29:45.0200748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70887
2023-01-11T22:29:45.0201430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0201878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0202448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0202913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0203537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0203996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0204570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0205030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0205470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0205958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0206610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0207275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0207799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0208267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0209538Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0210320Z   warnings.warn(
2023-01-11T22:29:45.0211458Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0212226Z   warnings.warn(
2023-01-11T22:29:45.0212954Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0213723Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0214533Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0215298Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0216074Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0216896Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0217748Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0218509Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0219299Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0220046Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0220838Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0221589Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0222382Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0223143Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0223921Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0224678Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0225465Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0226217Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0227003Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0227763Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0228529Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0229344Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0230173Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0230929Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0231713Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0232478Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0233245Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0234007Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0234793Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0235539Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0236330Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0237064Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0237850Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0238600Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0239615Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0240388Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0241154Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0241991Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0242774Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0243580Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0243915Z dist init r=1, world=2
2023-01-11T22:29:45.0244147Z dist init r=0, world=2
2023-01-11T22:29:45.0244386Z ok (4.413s)
2023-01-11T22:29:45.0244928Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70969
2023-01-11T22:29:45.0245537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70970
2023-01-11T22:29:45.0246161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0246612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0247185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0247644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0248220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0248658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0249228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0249681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0250128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0250621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0251259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0251946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0252468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0252939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0254190Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0254970Z   warnings.warn(
2023-01-11T22:29:45.0256133Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0256955Z   warnings.warn(
2023-01-11T22:29:45.0257685Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0258516Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0259338Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0260105Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0260899Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0261650Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0262441Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0263198Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0263973Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0264769Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0265564Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0266315Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0267107Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0267845Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0268633Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0269378Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0270166Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0270987Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0271797Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0272557Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0273346Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0274095Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0274887Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0275636Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0276409Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0277164Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0277952Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0278691Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0279639Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0280418Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0281191Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0281945Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0282731Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0283576Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0284423Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0285170Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0285953Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0286702Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0287489Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0288240Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0288546Z dist init r=1, world=2
2023-01-11T22:29:45.0288796Z dist init r=0, world=2
2023-01-11T22:29:45.0289032Z ok (4.414s)
2023-01-11T22:29:45.0289553Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71052
2023-01-11T22:29:45.0290181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71053
2023-01-11T22:29:45.0290805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0291254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0291810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0292279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0292858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0293295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0293844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0294308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0294757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0295232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0295887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0296572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0297089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0297539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0298797Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0299637Z   warnings.warn(
2023-01-11T22:29:45.0300845Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0301624Z   warnings.warn(
2023-01-11T22:29:45.0302339Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0303102Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0303903Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0304653Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0305437Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0306194Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0306965Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0307724Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0308509Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0309255Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0310041Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0310799Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0311570Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0312381Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0313199Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0313953Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0314745Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0315482Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0316273Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0317014Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0318072Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0318744Z   warnings.warn(
2023-01-11T22:29:45.0319668Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0320443Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0321241Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0321991Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0323044Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0323696Z   warnings.warn(
2023-01-11T22:29:45.0324656Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0325403Z   warnings.warn(
2023-01-11T22:29:45.0326364Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0327011Z   warnings.warn(
2023-01-11T22:29:45.0327776Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0328551Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0329355Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0330103Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0330902Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0331656Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0332434Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0333199Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0333988Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0334736Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0335532Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0336268Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0337058Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0337801Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0338587Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0339396Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0339704Z dist init r=1, world=2
2023-01-11T22:29:45.0339953Z dist init r=0, world=2
2023-01-11T22:29:45.0340190Z ok (4.512s)
2023-01-11T22:29:45.0340754Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71135
2023-01-11T22:29:45.0341390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71136
2023-01-11T22:29:45.0342007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0342461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0343021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0343494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0344070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0344510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0345063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0345526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0345974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0346478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0347118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0347800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0348318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0348773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0350045Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0350822Z   warnings.warn(
2023-01-11T22:29:45.0351985Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0352753Z   warnings.warn(
2023-01-11T22:29:45.0353476Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0354287Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0355129Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0355893Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0356689Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0357453Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0358225Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0358992Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0359936Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0360697Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0361490Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0362244Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0363021Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0363778Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0364599Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0365350Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0366143Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0366968Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0367759Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0368554Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0369635Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0370297Z   warnings.warn(
2023-01-11T22:29:45.0371001Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0371750Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0372792Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0373443Z   warnings.warn(
2023-01-11T22:29:45.0374163Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0374928Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0375963Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0376623Z   warnings.warn(
2023-01-11T22:29:45.0377582Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0378218Z   warnings.warn(
2023-01-11T22:29:45.0378936Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0379678Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0380464Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0381288Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0382119Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0382880Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0383657Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0384416Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0385209Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0385951Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0386736Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0387489Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0388264Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0389010Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0389796Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0390538Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0390842Z dist init r=1, world=2
2023-01-11T22:29:45.0391093Z dist init r=0, world=2
2023-01-11T22:29:45.0391329Z ok (4.412s)
2023-01-11T22:29:45.0391869Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71218
2023-01-11T22:29:45.0392479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71219
2023-01-11T22:29:45.0393095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0393542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0394123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0394632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0395215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0395660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0396212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0396715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0397176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0397670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0398312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0399002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0399708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0400190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0401459Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0402239Z   warnings.warn(
2023-01-11T22:29:45.0403397Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0404171Z   warnings.warn(
2023-01-11T22:29:45.0404896Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0405644Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0406442Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0407187Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0407985Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0408740Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0409533Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0410360Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0411205Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0411968Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0412757Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0413515Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0414288Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0415044Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0415829Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0416574Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0417366Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0418117Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0418892Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0419645Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0420438Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0421183Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0421971Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0422763Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0423583Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0424333Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0425121Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0425868Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0426643Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0427395Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0428183Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0428935Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0429724Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0430474Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0431251Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0432005Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0432795Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0433541Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0434328Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0435138Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0435445Z dist init r=1, world=2
2023-01-11T22:29:45.0435695Z dist init r=0, world=2
2023-01-11T22:29:45.0435935Z ok (4.513s)
2023-01-11T22:29:45.0436508Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71301
2023-01-11T22:29:45.0437140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71302
2023-01-11T22:29:45.0437760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0438213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0438777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0439239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0440046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0440471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0441046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0441505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0441954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0442430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0443083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0443770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0444288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0444738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0446000Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0446777Z   warnings.warn(
2023-01-11T22:29:45.0447932Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0448685Z   warnings.warn(
2023-01-11T22:29:45.0449396Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0450152Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0451041Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0451793Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0452647Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0453404Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0454198Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0454947Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0455732Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0456494Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0457294Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0458031Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0458946Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0459693Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0460484Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0461232Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0462004Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0462760Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0463550Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0464408Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0465244Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0485629Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0486529Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0487328Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0488133Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0488884Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0489669Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0490421Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0491204Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0491951Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0492721Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0493488Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0494278Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0495029Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0495808Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0496724Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0497562Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0498325Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0499107Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0499856Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0500179Z dist init r=1, world=2
2023-01-11T22:29:45.0500414Z dist init r=0, world=2
2023-01-11T22:29:45.0500652Z ok (4.412s)
2023-01-11T22:29:45.0501197Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71384
2023-01-11T22:29:45.0501805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71385
2023-01-11T22:29:45.0502464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0502914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0503496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0503948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0504525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0504965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0505516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0505981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0506431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0506927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0507571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0508259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0508774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0509243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0510500Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0511346Z   warnings.warn(
2023-01-11T22:29:45.0512513Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0513323Z   warnings.warn(
2023-01-11T22:29:45.0514067Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0514830Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0515598Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0516362Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0517163Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0517916Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0518705Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0519647Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0520455Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0521202Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0521988Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0522736Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0523505Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0524261Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0525123Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0525869Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0526712Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0527475Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0528243Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0528995Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0529776Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0530521Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0531317Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0532069Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0532837Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0533588Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0534370Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0535113Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0535904Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0536643Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0537429Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0538239Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0539063Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0539829Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0540601Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0541356Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0542137Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0542883Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0543664Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0544423Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0544726Z dist init r=1, world=2
2023-01-11T22:29:45.0544976Z dist init r=0, world=2
2023-01-11T22:29:45.0545213Z ok (4.412s)
2023-01-11T22:29:45.0545736Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71467
2023-01-11T22:29:45.0546352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71468
2023-01-11T22:29:45.0546982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0547437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0547994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0548453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0549023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0549445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0550011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0550468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0550917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0551396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0552113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0552803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0553323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0553820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0555101Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0555882Z   warnings.warn(
2023-01-11T22:29:45.0557095Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0557859Z   warnings.warn(
2023-01-11T22:29:45.0558569Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0559334Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0560337Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0561088Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0561888Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:401: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0562652Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0563430Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:403: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0564225Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0565024Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0565865Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0566649Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0567434Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0568230Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:407: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0568979Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0569758Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:408: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0570512Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0571290Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0572048Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0572828Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0573569Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0574352Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:414: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0575111Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0575889Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:416: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0576650Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0577435Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0578187Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0579034Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0579790Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0580608Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:425: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0581368Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0582159Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:427: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0582901Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0583691Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0584435Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0585219Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0585963Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0586754Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:437: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0587500Z   self.assertEqual(0, outer_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0588272Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:438: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0589029Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0589348Z dist init r=1, world=2
2023-01-11T22:29:45.0589599Z dist init r=0, world=2
2023-01-11T22:29:45.0589821Z ok (4.513s)
2023-01-11T22:29:45.0590254Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71550
2023-01-11T22:29:45.0590780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71551
2023-01-11T22:29:45.0591406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0591904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0592482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0592950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0593509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0593947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0594561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0595036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0595470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0595966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0596631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0597315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0597818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0598288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0598642Z dist init r=1, world=2
2023-01-11T22:29:45.0598874Z dist init r=0, world=2
2023-01-11T22:29:45.0599110Z ok (4.011s)
2023-01-11T22:29:45.0599886Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71629
2023-01-11T22:29:45.0600482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71630
2023-01-11T22:29:45.0601100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0601548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0602116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0602565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0603142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0603581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0604147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0604593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0605050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0605549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0606183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0606869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0607106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0607335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0608364Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0608565Z   warnings.warn(
2023-01-11T22:29:45.0609631Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0609750Z   warnings.warn(
2023-01-11T22:29:45.0609863Z dist init r=0, world=2
2023-01-11T22:29:45.0609972Z dist init r=1, world=2
2023-01-11T22:29:45.0610059Z ok (4.011s)
2023-01-11T22:29:45.0610446Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71708
2023-01-11T22:29:45.0610666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71709
2023-01-11T22:29:45.0611044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0611217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0611599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0611792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0612163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0612322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0612701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0612888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0613135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0613377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0613780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0614176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0614405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0614638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0615666Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0615765Z   warnings.warn(
2023-01-11T22:29:45.0616770Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0616938Z   warnings.warn(
2023-01-11T22:29:45.0617053Z dist init r=0, world=2
2023-01-11T22:29:45.0617162Z dist init r=1, world=2
2023-01-11T22:29:45.0617263Z ok (3.911s)
2023-01-11T22:29:45.0617652Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71787
2023-01-11T22:29:45.0617911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71788
2023-01-11T22:29:45.0618290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0618450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0618830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0619025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0619388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0619562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0619935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0620123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0620372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0620614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0620996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0621395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0621622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0621847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0622869Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0622986Z   warnings.warn(
2023-01-11T22:29:45.0623983Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0624095Z   warnings.warn(
2023-01-11T22:29:45.0624205Z dist init r=0, world=2
2023-01-11T22:29:45.0624314Z dist init r=1, world=2
2023-01-11T22:29:45.0624413Z ok (3.911s)
2023-01-11T22:29:45.0624784Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71866
2023-01-11T22:29:45.0625004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71867
2023-01-11T22:29:45.0625373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0625602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0625984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0626176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0626538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0626711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0627112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0627310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0627555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0627798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0628207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0628601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0628828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0629058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0630086Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0630203Z   warnings.warn(
2023-01-11T22:29:45.0631213Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0631325Z   warnings.warn(
2023-01-11T22:29:45.0631418Z dist init r=1, world=2
2023-01-11T22:29:45.0631527Z dist init r=0, world=2
2023-01-11T22:29:45.0631626Z ok (4.012s)
2023-01-11T22:29:45.0632017Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71945
2023-01-11T22:29:45.0632242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71946
2023-01-11T22:29:45.0632615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0632788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0633165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0633342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0633707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0633880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0634250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0634500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0634745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0634987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0635391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0635829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0636049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0636278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0637307Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0637426Z   warnings.warn(
2023-01-11T22:29:45.0638428Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0638538Z   warnings.warn(
2023-01-11T22:29:45.0638648Z dist init r=0, world=2
2023-01-11T22:29:45.0638761Z dist init r=1, world=2
2023-01-11T22:29:45.0638860Z ok (4.012s)
2023-01-11T22:29:45.0639249Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72024
2023-01-11T22:29:45.0639673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72025
2023-01-11T22:29:45.0640068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0640249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0640628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0640820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0641188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0641366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0641741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0641929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0642159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0642406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0642809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0643202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0643430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0643740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0644819Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0644943Z   warnings.warn(
2023-01-11T22:29:45.0645952Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0646066Z   warnings.warn(
2023-01-11T22:29:45.0646179Z dist init r=1, world=2
2023-01-11T22:29:45.0646272Z dist init r=0, world=2
2023-01-11T22:29:45.0646374Z ok (4.011s)
2023-01-11T22:29:45.0646759Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72103
2023-01-11T22:29:45.0646982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72104
2023-01-11T22:29:45.0647357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0647535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0647915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0648109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0648456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0648635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0649012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0649206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0649453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0649699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0650099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0650499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0650728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0650937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0651963Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0652079Z   warnings.warn(
2023-01-11T22:29:45.0653083Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0653252Z   warnings.warn(
2023-01-11T22:29:45.0653366Z dist init r=0, world=2
2023-01-11T22:29:45.0653477Z dist init r=1, world=2
2023-01-11T22:29:45.0653628Z ok (3.911s)
2023-01-11T22:29:45.0654021Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72182
2023-01-11T22:29:45.0654243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72183
2023-01-11T22:29:45.0654620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0654783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0655163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0655355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0655717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0655896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0656271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0656458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0656703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0656931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0657333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0657729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0657959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0658190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0659210Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0659328Z   warnings.warn(
2023-01-11T22:29:45.0660333Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0660444Z   warnings.warn(
2023-01-11T22:29:45.0660555Z dist init r=1, world=2
2023-01-11T22:29:45.0660663Z dist init r=0, world=2
2023-01-11T22:29:45.0660745Z ok (3.911s)
2023-01-11T22:29:45.0661096Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72261
2023-01-11T22:29:45.0661374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72262
2023-01-11T22:29:45.0661777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0661952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0662330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0662564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0662937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0663093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0663470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0663665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0663912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0664157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0664604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0665003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0665232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0665461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0665556Z dist init r=0, world=2
2023-01-11T22:29:45.0665667Z dist init r=1, world=2
2023-01-11T22:29:45.0665771Z ok (3.911s)
2023-01-11T22:29:45.0666117Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72340
2023-01-11T22:29:45.0666335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72341
2023-01-11T22:29:45.0666709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0666888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0667266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0667438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0667803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0667978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0668349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0668536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0668781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0669026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0669429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0669826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0670037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0670322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0670436Z dist init r=0, world=2
2023-01-11T22:29:45.0670545Z dist init r=1, world=2
2023-01-11T22:29:45.0670645Z ok (4.011s)
2023-01-11T22:29:45.0670964Z   test_summon_full_param_writeback (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72419
2023-01-11T22:29:45.0671183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72420
2023-01-11T22:29:45.0671605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0671768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0672153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0672346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0672712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0672884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0673261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0673450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0673698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0673922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0674325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0674718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0674949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0675177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0675938Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0676678Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0677422Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0678159Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0678883Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0679835Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0680683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0681470Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0682223Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0682956Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0683693Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0684419Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0685154Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0685876Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0686610Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0687339Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0688366Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1341: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:29:45.0688622Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2023-01-11T22:29:45.0689686Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1341: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:29:45.0689897Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2023-01-11T22:29:45.0690644Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0691383Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0692128Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0692858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0692980Z dist init r=0, world=2
2023-01-11T22:29:45.0693072Z dist init r=1, world=2
2023-01-11T22:29:45.0693172Z ok (4.110s)
2023-01-11T22:29:45.0693543Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72498
2023-01-11T22:29:45.0693763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72499
2023-01-11T22:29:45.0694143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0694318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0694698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0694891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0695238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0695415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0695784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0695972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0696218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0696465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0696866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0697260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0697550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0697761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0697874Z dist init r=0, world=2
2023-01-11T22:29:45.0697984Z dist init r=1, world=2
2023-01-11T22:29:45.0698084Z ok (4.011s)
2023-01-11T22:29:45.0698496Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72577
2023-01-11T22:29:45.0698726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72578
2023-01-11T22:29:45.0699103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0699277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0699636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0699838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0700206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0700380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0700751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0700942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0701188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0701429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0701827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0702209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0702441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0702669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0703512Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0703627Z   warnings.warn(
2023-01-11T22:29:45.0704455Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:792: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0704569Z   warnings.warn(
2023-01-11T22:29:45.0705392Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0705503Z   warnings.warn(
2023-01-11T22:29:45.0706321Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_unshard_param_utils.py:147: UserWarning: offload_to_cpu and rank0_only=False will result in full parameters being redundantly copied to CPU memory for GPUs that reside on the same machine, which may incur the risk of CPU OOM. It is recommended to use ``offload_to_cpu`` with rank0_only=True.
2023-01-11T22:29:45.0706489Z   warnings.warn(
2023-01-11T22:29:45.0706602Z dist init r=0, world=2
2023-01-11T22:29:45.0706694Z dist init r=1, world=2
2023-01-11T22:29:45.0706795Z ok (3.911s)
2023-01-11T22:29:45.0707163Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72656
2023-01-11T22:29:45.0707424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72657
2023-01-11T22:29:45.0707809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0707985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0708363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0708559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0708912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0709087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0709462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0709649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0709898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0710143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0710542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0710936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0711173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0711385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0711497Z dist init r=0, world=2
2023-01-11T22:29:45.0711606Z dist init r=1, world=2
2023-01-11T22:29:45.0711705Z ok (4.011s)
2023-01-11T22:29:45.0712079Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72735
2023-01-11T22:29:45.0712298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72736
2023-01-11T22:29:45.0712673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0712846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0713213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0713404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0713762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0713935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0714307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0714496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0714743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0714985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0715424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0715820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0716047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0716277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0716445Z dist init r=1, world=2
2023-01-11T22:29:45.0716565Z dist init r=0, world=2
2023-01-11T22:29:45.0716666Z ok (3.912s)
2023-01-11T22:29:45.0717019Z   test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72814
2023-01-11T22:29:45.0717237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72815
2023-01-11T22:29:45.0717595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0717771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0718150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0718340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0718706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0718881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0719253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0719654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0719891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0720142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0720549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0720944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0721177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0721403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0722427Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0722545Z   warnings.warn(
2023-01-11T22:29:45.0723553Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0723662Z   warnings.warn(
2023-01-11T22:29:45.0724251Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:271: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0724535Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0725161Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:271: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0725364Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0725932Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:273: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0726136Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0726712Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:273: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0726907Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0727472Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:279: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0727673Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0728261Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:279: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0728456Z   outer_full_param_size, outer_param._full_param_padded.storage().size()
2023-01-11T22:29:45.0729027Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:281: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0729224Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0729794Z /var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_summon_full_params.py:281: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:45.0729989Z   self.assertEqual(0, inner_param._full_param_padded.storage().size())
2023-01-11T22:29:45.0730103Z dist init r=0, world=2
2023-01-11T22:29:45.0730212Z dist init r=1, world=2
2023-01-11T22:29:45.0730294Z ok (4.511s)
2023-01-11T22:29:45.0730605Z   test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72893
2023-01-11T22:29:45.0730884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72894
2023-01-11T22:29:45.0731269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0731445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0731825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0732020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0732429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0732590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0732973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0733166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0733411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0733656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0734055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0734455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0734686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0734913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0735941Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0736061Z   warnings.warn(
2023-01-11T22:29:45.0737068Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:29:45.0737180Z   warnings.warn(
2023-01-11T22:29:45.0737292Z dist init r=1, world=2
2023-01-11T22:29:45.0737402Z dist init r=0, world=2
2023-01-11T22:29:45.0737504Z ok (4.011s)
2023-01-11T22:29:45.0737680Z   test_with_grads_core (__main__.TestSummonFullParams)
2023-01-11T22:29:45.0737986Z Tests the core usage of ``summon_full_params(with_grads=True)``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72972
2023-01-11T22:29:45.0738205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72973
2023-01-11T22:29:45.0738564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0738745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0739123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0739313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0739675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0739907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0740284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0740471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0740720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0740990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0741401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0741798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0742031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0742263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0742502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0742736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0742962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0743200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0743415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0743646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0743870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0744093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0744325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0744554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0744780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0745004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0745211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0745439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0745665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0745890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0746116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0746341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0746568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0746793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0747000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0747225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0747448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0747670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:29:45.0747782Z dist init r=1, world=2
2023-01-11T22:29:45.0747892Z dist init r=0, world=2
2023-01-11T22:29:45.0748053Z ok (7.015s)
2023-01-11T22:29:45.0748241Z   test_with_grads_none_grads (__main__.TestSummonFullParams)
2023-01-11T22:29:45.0748677Z Tests that if all ranks' ``FlatParameter`` has ``None`` gradient, then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73055
2023-01-11T22:29:45.0748897Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73056
2023-01-11T22:29:45.0749271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0749490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0749878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0750069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0750432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0750610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0750987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0751160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0751403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:45.0751650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0752051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0752443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:45.0752674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0752906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:45.0753017Z dist init r=1, world=2
2023-01-11T22:29:45.0753108Z dist init r=0, world=2
2023-01-11T22:29:45.0753208Z ok (4.010s)
2023-01-11T22:29:45.0753543Z   test_summon_full_param_writeback (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73134
2023-01-11T22:29:45.0753917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:29:45.0754092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:45.0754471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:45.0754660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:45.0754902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:45.0755305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:29:45.0755515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:45.0756050Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:29:45.0756167Z   warnings.warn(
2023-01-11T22:29:45.0756925Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0757672Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0758505Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0759257Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0760212Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0760970Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0761701Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:29:45.0761817Z dist init r=0, world=1
2023-01-11T22:29:45.0761921Z ok (3.809s)
2023-01-11T22:29:45.0761948Z 
2023-01-11T22:29:45.0762217Z ----------------------------------------------------------------------
2023-01-11T22:29:45.0762335Z Ran 52 tests in 215.756s
2023-01-11T22:29:45.0762355Z 
2023-01-11T22:29:45.0762430Z OK
2023-01-11T22:29:45.0762467Z 
2023-01-11T22:29:45.0762575Z Generating XML reports...
2023-01-11T22:29:45.0763049Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20230111222608.xml
2023-01-11T22:29:45.0763554Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20230111222608.xml
2023-01-11T22:29:45.0763575Z 
2023-01-11T22:29:45.0764048Z ##[endgroup]
2023-01-11T22:29:45.0764614Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_kixem_8r)
2023-01-11T22:29:45.0764634Z 
2023-01-11T22:29:45.0764919Z Running distributed/fsdp/test_fsdp_state_dict ... [2023-01-11 22:29:44.997382]
2023-01-11T22:29:45.0765398Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:29:44.997674]
2023-01-11T22:37:44.2475362Z 
2023-01-11T22:37:44.2478303Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict
2023-01-11T22:37:44.2479823Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_2awsfp6_)
2023-01-11T22:37:44.2480502Z 
2023-01-11T22:37:44.2480701Z Running tests...
2023-01-11T22:37:44.2482723Z ----------------------------------------------------------------------
2023-01-11T22:37:44.2483350Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2023-01-11T22:37:44.2484014Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2485089Z Tests that we can save a state_dict and load it into a blank model ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:37:44.2489537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73209
2023-01-11T22:37:44.2490578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73210
2023-01-11T22:37:44.2491814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2492312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2492912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2493359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2493952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2494402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2494967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2495428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2496246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2496788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2497641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2498666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2499198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2499667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2500680Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2501945Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2503187Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2504393Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2505081Z dist init r=0, world=2
2023-01-11T22:37:44.2505322Z dist init r=1, world=2
2023-01-11T22:37:44.2505566Z ok (5.639s)
2023-01-11T22:37:44.2506069Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2506981Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73288
2023-01-11T22:37:44.2507629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73289
2023-01-11T22:37:44.2508247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2508686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2509319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2509752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2510335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2510826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2511626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2512368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2512832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2513307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2513974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2514660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2515183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2515653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2515999Z dist init r=0, world=2
2023-01-11T22:37:44.2516263Z dist init r=1, world=2
2023-01-11T22:37:44.2516474Z ok (4.012s)
2023-01-11T22:37:44.2517006Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2517681Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73367
2023-01-11T22:37:44.2518218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73368
2023-01-11T22:37:44.2518831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2519987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2520599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2521070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2521654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2522104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2522687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2523136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2523603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2524089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2524985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2525808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2526317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2526781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2527141Z dist init r=1, world=2
2023-01-11T22:37:44.2527373Z dist init r=0, world=2
2023-01-11T22:37:44.2527643Z ok (3.911s)
2023-01-11T22:37:44.2528235Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2528912Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73446
2023-01-11T22:37:44.2529449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73447
2023-01-11T22:37:44.2530072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2530516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2531073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2531545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2532129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2532575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2533133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2533588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2534035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2534530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2535191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2535876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2536405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2536848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2537214Z dist init r=1, world=2
2023-01-11T22:37:44.2537461Z dist init r=0, world=2
2023-01-11T22:37:44.2537682Z ok (4.012s)
2023-01-11T22:37:44.2538201Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2538887Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73525
2023-01-11T22:37:44.2539417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73526
2023-01-11T22:37:44.2540028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2540482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2541052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2541524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2542090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2542606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2543274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2543999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2544439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2544987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2545655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2546345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2546845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2547317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2547667Z dist init r=1, world=2
2023-01-11T22:37:44.2547902Z dist init r=0, world=2
2023-01-11T22:37:44.2548137Z ok (4.012s)
2023-01-11T22:37:44.2548651Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2549342Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73604
2023-01-11T22:37:44.2549848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73605
2023-01-11T22:37:44.2550457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2550907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2551468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2551933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2552511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2552952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2553505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2553967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2554433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2555268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2556078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2556763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2557281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2557731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2558090Z dist init r=1, world=2
2023-01-11T22:37:44.2558342Z dist init r=0, world=2
2023-01-11T22:37:44.2558577Z ok (4.012s)
2023-01-11T22:37:44.2559072Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2560215Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73683
2023-01-11T22:37:44.2560856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73684
2023-01-11T22:37:44.2561459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2561910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2562544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2563020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2563587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2564031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2564598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2565063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2565495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2565985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2566636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2567306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2567825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2568292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2568643Z dist init r=1, world=2
2023-01-11T22:37:44.2568880Z dist init r=0, world=2
2023-01-11T22:37:44.2569119Z ok (3.912s)
2023-01-11T22:37:44.2569626Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2570291Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73762
2023-01-11T22:37:44.2570818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73763
2023-01-11T22:37:44.2571427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2571873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2572425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2572899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2573475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2573914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2574460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2574917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2575370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2575847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2576503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2577250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2577768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2578216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2578563Z dist init r=0, world=2
2023-01-11T22:37:44.2578812Z dist init r=1, world=2
2023-01-11T22:37:44.2579031Z ok (4.012s)
2023-01-11T22:37:44.2579587Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2580285Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73841
2023-01-11T22:37:44.2580804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73842
2023-01-11T22:37:44.2581398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2581844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2582413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2582876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2583436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2583876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2584446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2584891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2585345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2585833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2586487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2587152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2587671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2588139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2589135Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2590384Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2591602Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2592827Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2593502Z dist init r=1, world=2
2023-01-11T22:37:44.2593753Z dist init r=0, world=2
2023-01-11T22:37:44.2593974Z ok (4.112s)
2023-01-11T22:37:44.2594485Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2595211Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73920
2023-01-11T22:37:44.2595742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73921
2023-01-11T22:37:44.2596335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2596782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2597355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2597819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2598378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2598816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2599714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2600279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2600734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2601228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2601895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2602568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2603085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2603552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2603904Z dist init r=0, world=2
2023-01-11T22:37:44.2604136Z dist init r=1, world=2
2023-01-11T22:37:44.2604376Z ok (4.012s)
2023-01-11T22:37:44.2604936Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2605602Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73999
2023-01-11T22:37:44.2606129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74000
2023-01-11T22:37:44.2606738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2607183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2607738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2608201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2608777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2609215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2609767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2610329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2610778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2611251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2611906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2612643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2613170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2613618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2613960Z dist init r=0, world=2
2023-01-11T22:37:44.2614209Z dist init r=1, world=2
2023-01-11T22:37:44.2614435Z ok (4.012s)
2023-01-11T22:37:44.2614944Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2615625Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74078
2023-01-11T22:37:44.2616146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74079
2023-01-11T22:37:44.2616745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2617189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2617760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2618226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2618794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2619231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2619797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2620243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2620693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2621184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2621839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2622508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2623031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2623497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2623848Z dist init r=1, world=2
2023-01-11T22:37:44.2624082Z dist init r=0, world=2
2023-01-11T22:37:44.2624318Z ok (4.011s)
2023-01-11T22:37:44.2624832Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2625497Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74157
2023-01-11T22:37:44.2626020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74158
2023-01-11T22:37:44.2626631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2627140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2627697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2628165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2628739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2629204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2629778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2630236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2630684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2631161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2631808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2632490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2633004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2633456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2633802Z dist init r=1, world=2
2023-01-11T22:37:44.2634052Z dist init r=0, world=2
2023-01-11T22:37:44.2634271Z ok (4.013s)
2023-01-11T22:37:44.2634776Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2635453Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74236
2023-01-11T22:37:44.2635975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74237
2023-01-11T22:37:44.2636565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2637008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2637586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2638034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2638610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2639047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2640254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2640699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2641148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2641637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2642300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2642971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2643488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2643954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2644388Z dist init r=1, world=2
2023-01-11T22:37:44.2644637Z dist init r=0, world=2
2023-01-11T22:37:44.2644877Z ok (3.911s)
2023-01-11T22:37:44.2645372Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2646106Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74315
2023-01-11T22:37:44.2646641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74316
2023-01-11T22:37:44.2647255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2647682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2648253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2648721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2649295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2649718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2650287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2650752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2651199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2651678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2652332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2653019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2653521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2653987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2654330Z dist init r=0, world=2
2023-01-11T22:37:44.2654579Z dist init r=1, world=2
2023-01-11T22:37:44.2654804Z ok (4.012s)
2023-01-11T22:37:44.2655314Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2655994Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74394
2023-01-11T22:37:44.2656504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74395
2023-01-11T22:37:44.2657113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2657560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2658128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2658580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2659159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2659598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2660167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2660675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2661127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2661620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2662264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2662986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2663513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2663976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2664312Z dist init r=1, world=2
2023-01-11T22:37:44.2664559Z dist init r=0, world=2
2023-01-11T22:37:44.2664801Z ok (3.911s)
2023-01-11T22:37:44.2665305Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2665998Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74473
2023-01-11T22:37:44.2666520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74474
2023-01-11T22:37:44.2667135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2667567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2668140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2668605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2669187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2669608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2670172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2670631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2671066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2671558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2672207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2672885Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2673391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2673854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2674852Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2676095Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2677332Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2678652Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2679249Z dist init r=1, world=2
2023-01-11T22:37:44.2679951Z dist init r=0, world=2
2023-01-11T22:37:44.2680191Z ok (4.012s)
2023-01-11T22:37:44.2680695Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2681390Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74552
2023-01-11T22:37:44.2681914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74553
2023-01-11T22:37:44.2682531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2682963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2683544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2684012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2684590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2685012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2685584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2686046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2686479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2686972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2687630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2688318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2688819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2689287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2689645Z dist init r=1, world=2
2023-01-11T22:37:44.2689894Z dist init r=0, world=2
2023-01-11T22:37:44.2690116Z ok (4.012s)
2023-01-11T22:37:44.2690630Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2691316Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74631
2023-01-11T22:37:44.2691822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74632
2023-01-11T22:37:44.2692435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2692880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2693448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2693995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2694577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2695019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2695570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2696089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2696550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2697042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2697680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2698366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2698880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2699343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2699679Z dist init r=0, world=2
2023-01-11T22:37:44.2699927Z dist init r=1, world=2
2023-01-11T22:37:44.2700167Z ok (3.913s)
2023-01-11T22:37:44.2700666Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2701357Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74710
2023-01-11T22:37:44.2701876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74711
2023-01-11T22:37:44.2702484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2702910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2703478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2703942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2704569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2704996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2705559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2706017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2706454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2706944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2707595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2708280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2708783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2709248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2709598Z dist init r=0, world=2
2023-01-11T22:37:44.2709845Z dist init r=1, world=2
2023-01-11T22:37:44.2710083Z ok (3.911s)
2023-01-11T22:37:44.2710603Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2711366Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74789
2023-01-11T22:37:44.2711873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74790
2023-01-11T22:37:44.2712526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2712978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2713557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2714008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2714578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2715023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2715572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2716032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2716478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2716972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2717607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2718288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2718809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2719274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2720705Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2721955Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2723179Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2724412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2725010Z dist init r=0, world=2
2023-01-11T22:37:44.2725248Z dist init r=1, world=2
2023-01-11T22:37:44.2725486Z ok (4.112s)
2023-01-11T22:37:44.2726000Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2726690Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74868
2023-01-11T22:37:44.2727292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74869
2023-01-11T22:37:44.2727903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2728345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2728916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2729427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2730017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2730458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2731007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2731479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2731928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2732417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2733055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2733745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2734261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2734728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2735060Z dist init r=1, world=2
2023-01-11T22:37:44.2735313Z dist init r=0, world=2
2023-01-11T22:37:44.2735551Z ok (4.011s)
2023-01-11T22:37:44.2736050Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2736733Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74947
2023-01-11T22:37:44.2737254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74948
2023-01-11T22:37:44.2737869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2738301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2738869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2739335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2739896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2740335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2740902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2741360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2741799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2742290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2742951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2743631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2744194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2744663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2745015Z dist init r=1, world=2
2023-01-11T22:37:44.2745245Z dist init r=0, world=2
2023-01-11T22:37:44.2745482Z ok (3.911s)
2023-01-11T22:37:44.2746038Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2746732Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75026
2023-01-11T22:37:44.2747235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75027
2023-01-11T22:37:44.2747851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2748298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2748854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2749318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2749900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2750340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2750888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2751350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2751799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2752302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2752938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2753616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2754138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2754586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2754938Z dist init r=1, world=2
2023-01-11T22:37:44.2755187Z dist init r=0, world=2
2023-01-11T22:37:44.2755427Z ok (3.913s)
2023-01-11T22:37:44.2755932Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2756620Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75105
2023-01-11T22:37:44.2757140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75106
2023-01-11T22:37:44.2757731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2758183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2758741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2759180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2760194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2760762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2761348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2761809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2762242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2762791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2763459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2764129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2764647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2765119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2766115Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2767355Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2768570Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2769807Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2770409Z dist init r=0, world=2
2023-01-11T22:37:44.2770660Z dist init r=1, world=2
2023-01-11T22:37:44.2770900Z ok (4.012s)
2023-01-11T22:37:44.2771401Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2772085Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75184
2023-01-11T22:37:44.2772613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75185
2023-01-11T22:37:44.2773205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2773650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2774219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2774687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2775242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2775685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2776253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2776777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2777208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2777699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2778355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2779097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2779628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2780100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2780449Z dist init r=1, world=2
2023-01-11T22:37:44.2780682Z dist init r=0, world=2
2023-01-11T22:37:44.2780917Z ok (4.112s)
2023-01-11T22:37:44.2781438Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2782112Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75263
2023-01-11T22:37:44.2782635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75264
2023-01-11T22:37:44.2783248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2783693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2784249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2784714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2785293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2785732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2786285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2786745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2787198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2787730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2788396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2789080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2789604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2790054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2790397Z dist init r=1, world=2
2023-01-11T22:37:44.2790646Z dist init r=0, world=2
2023-01-11T22:37:44.2790866Z ok (3.911s)
2023-01-11T22:37:44.2791385Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2792068Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75342
2023-01-11T22:37:44.2792588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75343
2023-01-11T22:37:44.2793179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2793694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2794267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2794731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2795288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2795821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2796407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2796850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2797301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2797795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2798448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2799115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2800030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2800510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2800862Z dist init r=0, world=2
2023-01-11T22:37:44.2801095Z dist init r=1, world=2
2023-01-11T22:37:44.2801333Z ok (4.012s)
2023-01-11T22:37:44.2801849Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2802526Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75421
2023-01-11T22:37:44.2803047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75422
2023-01-11T22:37:44.2803663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2804107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2804707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2805175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2805755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2806196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2806756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2807215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2807663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2808139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2808793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2809477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2809994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2810442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2812279Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2813594Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2814847Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2816082Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2816691Z dist init r=1, world=2
2023-01-11T22:37:44.2816925Z dist init r=0, world=2
2023-01-11T22:37:44.2817164Z ok (4.012s)
2023-01-11T22:37:44.2817684Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2818358Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75500
2023-01-11T22:37:44.2818879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75501
2023-01-11T22:37:44.2819497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2819944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2820501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2820970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2821551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2821992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2822541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2823002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2823454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2823928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2824580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2825265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2825787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2826236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2826578Z dist init r=0, world=2
2023-01-11T22:37:44.2826827Z dist init r=1, world=2
2023-01-11T22:37:44.2827048Z ok (4.011s)
2023-01-11T22:37:44.2827562Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2828331Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75579
2023-01-11T22:37:44.2828848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75580
2023-01-11T22:37:44.2829443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2829934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2830514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2830973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2831534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2831981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2832549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2832990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2833436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2833931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2834580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2835246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2835764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2836236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2836590Z dist init r=0, world=2
2023-01-11T22:37:44.2836822Z dist init r=1, world=2
2023-01-11T22:37:44.2837062Z ok (4.011s)
2023-01-11T22:37:44.2837576Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2838250Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75658
2023-01-11T22:37:44.2838773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75659
2023-01-11T22:37:44.2839760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2840290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2840855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2841314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2841890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2842315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2842887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2843346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2843795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2844268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2845022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2845709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2846228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2846678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2847092Z dist init r=0, world=2
2023-01-11T22:37:44.2847354Z dist init r=1, world=2
2023-01-11T22:37:44.2847575Z ok (4.012s)
2023-01-11T22:37:44.2848082Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2848763Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75737
2023-01-11T22:37:44.2849291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75738
2023-01-11T22:37:44.2849886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2850339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2850912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2851376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2851932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2852374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2852941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2853385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2853837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2854327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2854982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2855653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2856165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2856631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2856970Z dist init r=0, world=2
2023-01-11T22:37:44.2857218Z dist init r=1, world=2
2023-01-11T22:37:44.2857456Z ok (4.011s)
2023-01-11T22:37:44.2857961Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2858619Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75816
2023-01-11T22:37:44.2859143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75817
2023-01-11T22:37:44.2859754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2860199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2860754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2861283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2861861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2862285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2862854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2863311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2863808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2864296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2864953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2865646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2866162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2866611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2866959Z dist init r=0, world=2
2023-01-11T22:37:44.2867207Z dist init r=1, world=2
2023-01-11T22:37:44.2867429Z ok (4.012s)
2023-01-11T22:37:44.2867936Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2868617Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75895
2023-01-11T22:37:44.2869139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75896
2023-01-11T22:37:44.2869731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2870175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2870744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2871197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2871773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2872212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2872778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2873224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2873677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2874167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2874819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2875485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2876009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2876477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2877459Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2878781Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2880564Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2881813Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2882421Z dist init r=0, world=2
2023-01-11T22:37:44.2882671Z dist init r=1, world=2
2023-01-11T22:37:44.2882892Z ok (4.012s)
2023-01-11T22:37:44.2883396Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2884078Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75974
2023-01-11T22:37:44.2884597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75975
2023-01-11T22:37:44.2885191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2885636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2886217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2886669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2887248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2887685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2888257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2888701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2889148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2889637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2890272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2890967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2891487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2891956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2892292Z dist init r=0, world=2
2023-01-11T22:37:44.2892540Z dist init r=1, world=2
2023-01-11T22:37:44.2892781Z ok (4.012s)
2023-01-11T22:37:44.2893271Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2893946Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76053
2023-01-11T22:37:44.2894548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76054
2023-01-11T22:37:44.2895162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2895591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2896161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2896668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2915317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2915791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2916360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2916843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2917297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2917783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2918445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2919140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2919958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2920414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2920763Z dist init r=0, world=2
2023-01-11T22:37:44.2921015Z dist init r=1, world=2
2023-01-11T22:37:44.2921243Z ok (4.112s)
2023-01-11T22:37:44.2921753Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2922437Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76132
2023-01-11T22:37:44.2922967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76133
2023-01-11T22:37:44.2923578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2924035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2924612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2925081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2925651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2926089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2926657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2927101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2927559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2928059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2928716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2929382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2930061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2930535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2930893Z dist init r=1, world=2
2023-01-11T22:37:44.2931129Z dist init r=0, world=2
2023-01-11T22:37:44.2931367Z ok (4.012s)
2023-01-11T22:37:44.2931940Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2932614Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76211
2023-01-11T22:37:44.2933136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76212
2023-01-11T22:37:44.2933746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2934199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2934758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2935227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2935806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2936251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2936799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2937260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2937712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2938199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2938854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2939537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2940055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2940510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2941504Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2942737Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2943976Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2944584Z dist init r=1, world=2
2023-01-11T22:37:44.2944821Z dist init r=0, world=2
2023-01-11T22:37:44.2945066Z ok (4.013s)
2023-01-11T22:37:44.2945576Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2946325Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76290
2023-01-11T22:37:44.2946833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76291
2023-01-11T22:37:44.2947447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2947897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2948505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2948988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2949570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2950012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2950569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2951027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2951482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2951981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2952624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2953311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2953835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2954290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2954649Z dist init r=0, world=2
2023-01-11T22:37:44.2954896Z dist init r=1, world=2
2023-01-11T22:37:44.2955120Z ok (4.112s)
2023-01-11T22:37:44.2955632Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2956311Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76369
2023-01-11T22:37:44.2956833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76370
2023-01-11T22:37:44.2957422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2957872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2958443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2958917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2959672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2960144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2960708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2961178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2961615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2962112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2962766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2963527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2964046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2964517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2964870Z dist init r=0, world=2
2023-01-11T22:37:44.2965104Z dist init r=1, world=2
2023-01-11T22:37:44.2965343Z ok (4.012s)
2023-01-11T22:37:44.2965914Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2966585Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76448
2023-01-11T22:37:44.2967109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76449
2023-01-11T22:37:44.2967723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2968173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2968728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2969195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2969776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2970211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2970755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2971219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2971676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2972155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2972808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2973498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2974022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2974470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2974817Z dist init r=1, world=2
2023-01-11T22:37:44.2975070Z dist init r=0, world=2
2023-01-11T22:37:44.2975291Z ok (4.012s)
2023-01-11T22:37:44.2975801Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2976474Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76527
2023-01-11T22:37:44.2977000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76528
2023-01-11T22:37:44.2977599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2978046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2978622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2979090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2979722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2980158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2980723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2981166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2981621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2982174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2982841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2983504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2984033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2984500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.2985502Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2986728Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2987969Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2989205Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.2989804Z dist init r=0, world=2
2023-01-11T22:37:44.2990060Z dist init r=1, world=2
2023-01-11T22:37:44.2990285Z ok (4.011s)
2023-01-11T22:37:44.2990789Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.2991465Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76606
2023-01-11T22:37:44.2991993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76607
2023-01-11T22:37:44.2992585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2993034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2993613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2994082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2994644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.2995084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.2995650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.2996160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.2996616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.2997116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.2997770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2998488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.2999016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.2999662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3000036Z dist init r=0, world=2
2023-01-11T22:37:44.3000274Z dist init r=1, world=2
2023-01-11T22:37:44.3000504Z ok (4.112s)
2023-01-11T22:37:44.3001012Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3001674Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76685
2023-01-11T22:37:44.3002202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76686
2023-01-11T22:37:44.3002826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3003279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3003835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3004308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3004925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3005347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3005914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3006372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3006829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3007308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3007954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3008636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3009164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3009616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3009961Z dist init r=1, world=2
2023-01-11T22:37:44.3010212Z dist init r=0, world=2
2023-01-11T22:37:44.3010436Z ok (4.112s)
2023-01-11T22:37:44.3010949Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3011631Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76764
2023-01-11T22:37:44.3012159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76765
2023-01-11T22:37:44.3012860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3013307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3013880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3014332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3014974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3015421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3015997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3016441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3016894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3017397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3018056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3018729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3019248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3019720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3020057Z dist init r=0, world=2
2023-01-11T22:37:44.3020308Z dist init r=1, world=2
2023-01-11T22:37:44.3020548Z ok (4.011s)
2023-01-11T22:37:44.3021033Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3021707Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76843
2023-01-11T22:37:44.3022230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76844
2023-01-11T22:37:44.3022838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3023268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3023839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3024306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3024884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3025310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3025880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3026341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3026800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3027285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3027945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3028638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3029136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3029673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3030672Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3031971Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3032594Z dist init r=0, world=2
2023-01-11T22:37:44.3032827Z dist init r=1, world=2
2023-01-11T22:37:44.3033064Z ok (4.011s)
2023-01-11T22:37:44.3033567Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3034250Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76922
2023-01-11T22:37:44.3034755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76923
2023-01-11T22:37:44.3035369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3035823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3036385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3036853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3037432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3037880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3038435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3038901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3039352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3040034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3040696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3041379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3041898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3042353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3042708Z dist init r=0, world=2
2023-01-11T22:37:44.3042961Z dist init r=1, world=2
2023-01-11T22:37:44.3043186Z ok (4.112s)
2023-01-11T22:37:44.3043658Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77001
2023-01-11T22:37:44.3044220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77002
2023-01-11T22:37:44.3044829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3045258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3045832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3046393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3046983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3047706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3048290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3048752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3049277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3049787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3050446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3051138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3051642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3052108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3053379Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3054160Z   warnings.warn(
2023-01-11T22:37:44.3055313Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3056057Z   warnings.warn(
2023-01-11T22:37:44.3056305Z dist init r=1, world=2
2023-01-11T22:37:44.3056558Z dist init r=0, world=2
2023-01-11T22:37:44.3056662Z ok (4.010s)
2023-01-11T22:37:44.3057016Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77080
2023-01-11T22:37:44.3057221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77081
2023-01-11T22:37:44.3057604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3057785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3058165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3058358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3058723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3058899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3059274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3059445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3059699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3059943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3060416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3060813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3061042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3061317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3062348Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3062468Z   warnings.warn(
2023-01-11T22:37:44.3063469Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3063581Z   warnings.warn(
2023-01-11T22:37:44.3063693Z dist init r=1, world=2
2023-01-11T22:37:44.3063786Z dist init r=0, world=2
2023-01-11T22:37:44.3063885Z ok (4.010s)
2023-01-11T22:37:44.3064224Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77159
2023-01-11T22:37:44.3064449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77160
2023-01-11T22:37:44.3064823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3065001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3065382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3065575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3065928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3066106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3066483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3066672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3066923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3067169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3067569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3067969Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3068202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3068413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3069439Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3069613Z   warnings.warn(
2023-01-11T22:37:44.3070662Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3070779Z   warnings.warn(
2023-01-11T22:37:44.3070891Z dist init r=0, world=2
2023-01-11T22:37:44.3071002Z dist init r=1, world=2
2023-01-11T22:37:44.3071105Z ok (4.011s)
2023-01-11T22:37:44.3071450Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3071903Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77238
2023-01-11T22:37:44.3072106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77239
2023-01-11T22:37:44.3072479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3072654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3073033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3073224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3073600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3073775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3074149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3074334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3074562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3074809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3075211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3075608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3075846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3076077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3076699Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3076815Z   warnings.warn(
2023-01-11T22:37:44.3077439Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3077534Z   warnings.warn(
2023-01-11T22:37:44.3077649Z dist init r=0, world=2
2023-01-11T22:37:44.3077759Z dist init r=1, world=2
2023-01-11T22:37:44.3077860Z ok (4.011s)
2023-01-11T22:37:44.3078265Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3078715Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77317
2023-01-11T22:37:44.3078936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77318
2023-01-11T22:37:44.3079674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3079865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3080249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3080442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3080816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3080990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3081365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3081555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3081805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3082058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3082442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3082839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3083076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3083304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3083930Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3084045Z   warnings.warn(
2023-01-11T22:37:44.3084669Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3084781Z   warnings.warn(
2023-01-11T22:37:44.3084894Z dist init r=0, world=2
2023-01-11T22:37:44.3084986Z dist init r=1, world=2
2023-01-11T22:37:44.3085087Z ok (4.011s)
2023-01-11T22:37:44.3085418Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3085868Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77396
2023-01-11T22:37:44.3086088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77397
2023-01-11T22:37:44.3086461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3086638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3087021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3087214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3087568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3087887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3088269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3088457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3088704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3089008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3089420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3089811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3090023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3090257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3090878Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3090990Z   warnings.warn(
2023-01-11T22:37:44.3091612Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3091724Z   warnings.warn(
2023-01-11T22:37:44.3091837Z dist init r=1, world=2
2023-01-11T22:37:44.3091944Z dist init r=0, world=2
2023-01-11T22:37:44.3092045Z ok (4.011s)
2023-01-11T22:37:44.3092355Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3092810Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77475
2023-01-11T22:37:44.3093031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77476
2023-01-11T22:37:44.3093401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3093580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3093961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3094153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3094521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3094698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3095055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3095244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3095491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3095738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3096139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3096536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3096766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3097053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3097677Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3097773Z   warnings.warn(
2023-01-11T22:37:44.3098423Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3098540Z   warnings.warn(
2023-01-11T22:37:44.3098653Z dist init r=0, world=2
2023-01-11T22:37:44.3098764Z dist init r=1, world=2
2023-01-11T22:37:44.3098867Z ok (4.011s)
2023-01-11T22:37:44.3099203Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3099655Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77554
2023-01-11T22:37:44.3099857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77555
2023-01-11T22:37:44.3100231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3100411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3100794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3100985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3101352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3101530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3101905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3102098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3102326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3102575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3102977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3103374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3103604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3103837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3104461Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3104575Z   warnings.warn(
2023-01-11T22:37:44.3105244Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3105337Z   warnings.warn(
2023-01-11T22:37:44.3105450Z dist init r=1, world=2
2023-01-11T22:37:44.3105561Z dist init r=0, world=2
2023-01-11T22:37:44.3105662Z ok (4.011s)
2023-01-11T22:37:44.3105993Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3106522Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77633
2023-01-11T22:37:44.3106743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77634
2023-01-11T22:37:44.3107113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3107329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3107722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3107914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3108283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3108462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3108832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3109021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3109270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3109513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3109899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3110298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3110529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3110757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3111380Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3111492Z   warnings.warn(
2023-01-11T22:37:44.3112113Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3112225Z   warnings.warn(
2023-01-11T22:37:44.3112337Z dist init r=0, world=2
2023-01-11T22:37:44.3112429Z dist init r=1, world=2
2023-01-11T22:37:44.3112530Z ok (4.011s)
2023-01-11T22:37:44.3112874Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3113326Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77712
2023-01-11T22:37:44.3113545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77713
2023-01-11T22:37:44.3113915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3114089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3114473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3114667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3115017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3115250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3115631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3115820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3116066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3116310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3116755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3117163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3117392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3117607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3118236Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3118349Z   warnings.warn(
2023-01-11T22:37:44.3118964Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3119073Z   warnings.warn(
2023-01-11T22:37:44.3119183Z dist init r=0, world=2
2023-01-11T22:37:44.3119293Z dist init r=1, world=2
2023-01-11T22:37:44.3119636Z ok (4.013s)
2023-01-11T22:37:44.3119981Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3120455Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77791
2023-01-11T22:37:44.3120668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77792
2023-01-11T22:37:44.3121039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3121222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3121605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3121799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3122170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3122342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3122703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3122889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3123135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3123381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3123787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3124183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3124415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3124643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3125359Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3125454Z   warnings.warn(
2023-01-11T22:37:44.3126130Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3126251Z   warnings.warn(
2023-01-11T22:37:44.3126364Z dist init r=0, world=2
2023-01-11T22:37:44.3126475Z dist init r=1, world=2
2023-01-11T22:37:44.3126575Z ok (4.011s)
2023-01-11T22:37:44.3126910Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3127364Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77870
2023-01-11T22:37:44.3127585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77871
2023-01-11T22:37:44.3127937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3128112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3128494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3128686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3129053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3129230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3129614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3129806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3130035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3130280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3130683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3131080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3131317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3131546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3132174Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3132289Z   warnings.warn(
2023-01-11T22:37:44.3132912Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3133024Z   warnings.warn(
2023-01-11T22:37:44.3133119Z dist init r=1, world=2
2023-01-11T22:37:44.3133229Z dist init r=0, world=2
2023-01-11T22:37:44.3133329Z ok (4.011s)
2023-01-11T22:37:44.3133662Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3134171Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77949
2023-01-11T22:37:44.3134390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77950
2023-01-11T22:37:44.3134757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3134932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3135339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3135538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3135908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3136081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3136462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3136654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3136900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3137146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3137532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3137932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3138164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3138391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3139017Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3139133Z   warnings.warn(
2023-01-11T22:37:44.3139751Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3139868Z   warnings.warn(
2023-01-11T22:37:44.3139982Z dist init r=0, world=2
2023-01-11T22:37:44.3140074Z dist init r=1, world=2
2023-01-11T22:37:44.3140176Z ok (4.011s)
2023-01-11T22:37:44.3140509Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3140958Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78028
2023-01-11T22:37:44.3141181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78029
2023-01-11T22:37:44.3141550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3141725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3142108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3142302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3142646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3142819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3143189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3143438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3143684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3143926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3144331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3144769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3145006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3145216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3145334Z dist init r=1, world=2
2023-01-11T22:37:44.3145443Z dist init r=0, world=2
2023-01-11T22:37:44.3145539Z ok (4.011s)
2023-01-11T22:37:44.3145869Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3146324Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78107
2023-01-11T22:37:44.3146547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78108
2023-01-11T22:37:44.3146922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3147079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3147457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3147653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3148020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3148190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3148560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3148747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3148996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3149223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3149627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3150023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3150258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3150481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3150592Z dist init r=0, world=2
2023-01-11T22:37:44.3150697Z dist init r=1, world=2
2023-01-11T22:37:44.3150796Z ok (3.911s)
2023-01-11T22:37:44.3151104Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3151554Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78186
2023-01-11T22:37:44.3151770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78187
2023-01-11T22:37:44.3152143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3152374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3152755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3152946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3153311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3153526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3153888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3154073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3154316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3154562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3154954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3155339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3155565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3155790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3155893Z dist init r=1, world=2
2023-01-11T22:37:44.3155986Z dist init r=0, world=2
2023-01-11T22:37:44.3156080Z ok (4.013s)
2023-01-11T22:37:44.3156389Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3156832Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78265
2023-01-11T22:37:44.3157041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78266
2023-01-11T22:37:44.3157405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3157577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3157956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3158129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3158499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3158674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3159057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3159247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3159729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3159977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3160383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3160777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3160990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3161213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3161411Z dist init r=0, world=2
2023-01-11T22:37:44.3161521Z dist init r=1, world=2
2023-01-11T22:37:44.3161623Z ok (4.111s)
2023-01-11T22:37:44.3161946Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3162398Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78344
2023-01-11T22:37:44.3162673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78345
2023-01-11T22:37:44.3163039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3163214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3163589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3163782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3164147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3164320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3164691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3164879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3165126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3165355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3165756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3166155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3166383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3166611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3166723Z dist init r=0, world=2
2023-01-11T22:37:44.3166833Z dist init r=1, world=2
2023-01-11T22:37:44.3166935Z ok (4.111s)
2023-01-11T22:37:44.3167239Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3167689Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78423
2023-01-11T22:37:44.3167907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78424
2023-01-11T22:37:44.3168280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3168457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3168834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3169022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3169389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3169563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3169918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3170103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3170345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3170646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3171049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3171443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3171728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3171961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3172075Z dist init r=0, world=2
2023-01-11T22:37:44.3172167Z dist init r=1, world=2
2023-01-11T22:37:44.3172268Z ok (4.011s)
2023-01-11T22:37:44.3172609Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3173064Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78502
2023-01-11T22:37:44.3173284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78503
2023-01-11T22:37:44.3173654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3173825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3174207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3174379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3174748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3174924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3175300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3175487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3175733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3175980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3176382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3176778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3176989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3177220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3177337Z dist init r=0, world=2
2023-01-11T22:37:44.3177447Z dist init r=1, world=2
2023-01-11T22:37:44.3177549Z ok (4.011s)
2023-01-11T22:37:44.3177887Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3178341Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78581
2023-01-11T22:37:44.3178559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78582
2023-01-11T22:37:44.3178913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3179087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3179462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3179710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3180080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3180253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3180626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3180860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3181110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3181337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3181738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3182140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3182369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3182596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3182709Z dist init r=0, world=2
2023-01-11T22:37:44.3182821Z dist init r=1, world=2
2023-01-11T22:37:44.3182922Z ok (4.012s)
2023-01-11T22:37:44.3183232Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3183682Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78660
2023-01-11T22:37:44.3183901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78661
2023-01-11T22:37:44.3184275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3184449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3184826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3185017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3185387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3185563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3185918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3186107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3186356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3186597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3186998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3187391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3187623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3187853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3187947Z dist init r=1, world=2
2023-01-11T22:37:44.3188055Z dist init r=0, world=2
2023-01-11T22:37:44.3188156Z ok (4.011s)
2023-01-11T22:37:44.3188481Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3188996Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78739
2023-01-11T22:37:44.3189214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78740
2023-01-11T22:37:44.3189583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3189803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3190193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3190366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3190735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3190911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3191284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3191471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3191714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3191961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3192359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3192736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3192967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3193197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3193310Z dist init r=0, world=2
2023-01-11T22:37:44.3193420Z dist init r=1, world=2
2023-01-11T22:37:44.3193520Z ok (4.011s)
2023-01-11T22:37:44.3193847Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3194171Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78818
2023-01-11T22:37:44.3194391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78819
2023-01-11T22:37:44.3194746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3194925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3195312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3195505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3195872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3196045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3196418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3196604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3196831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3197074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3197473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3197934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3198165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3198391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3198504Z dist init r=1, world=2
2023-01-11T22:37:44.3198655Z dist init r=0, world=2
2023-01-11T22:37:44.3198743Z ok (4.513s)
2023-01-11T22:37:44.3199072Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3199613Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78901
2023-01-11T22:37:44.3199856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78902
2023-01-11T22:37:44.3200232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3200410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3200786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3200982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3201349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3201505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3201879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3202073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3202321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3202561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3202962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3203364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3203593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3203822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3203916Z dist init r=1, world=2
2023-01-11T22:37:44.3204027Z dist init r=0, world=2
2023-01-11T22:37:44.3204128Z ok (4.012s)
2023-01-11T22:37:44.3204463Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3204822Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78980
2023-01-11T22:37:44.3205043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78981
2023-01-11T22:37:44.3205419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3205598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3205960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3206154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3206521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3206784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3207163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3207352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3207597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3207896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3208308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3208685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3208920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3209150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3209263Z dist init r=0, world=2
2023-01-11T22:37:44.3209376Z dist init r=1, world=2
2023-01-11T22:37:44.3209477Z ok (4.613s)
2023-01-11T22:37:44.3209802Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3210122Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79063
2023-01-11T22:37:44.3210325Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79064
2023-01-11T22:37:44.3210697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3210878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3211257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3211441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3211798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3211972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3212347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3212536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3212765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3213010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3213413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3213804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3214030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3214255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3214372Z dist init r=0, world=2
2023-01-11T22:37:44.3214482Z dist init r=1, world=2
2023-01-11T22:37:44.3214565Z ok (3.912s)
2023-01-11T22:37:44.3214898Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3215220Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79142
2023-01-11T22:37:44.3215496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79143
2023-01-11T22:37:44.3215871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3216046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3216425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3216656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3217030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3217187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3217559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3217751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3217992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3218236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3218633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3219034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3219264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3219491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3219585Z dist init r=1, world=2
2023-01-11T22:37:44.3219699Z dist init r=0, world=2
2023-01-11T22:37:44.3219799Z ok (4.513s)
2023-01-11T22:37:44.3220130Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3220447Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79225
2023-01-11T22:37:44.3220663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79226
2023-01-11T22:37:44.3221041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3221217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3221579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3221770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3222143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3222318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3222689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3222878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3223127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3223372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3223751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3224148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3224473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3224701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3224815Z dist init r=1, world=2
2023-01-11T22:37:44.3224926Z dist init r=0, world=2
2023-01-11T22:37:44.3225029Z ok (3.912s)
2023-01-11T22:37:44.3225404Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3225730Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79304
2023-01-11T22:37:44.3225932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79305
2023-01-11T22:37:44.3226312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3226492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3226872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3227065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3227429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3227607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3227982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3228150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3228395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3228639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3229034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3229429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3229659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3229890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3230003Z dist init r=1, world=2
2023-01-11T22:37:44.3230113Z dist init r=0, world=2
2023-01-11T22:37:44.3230196Z ok (4.513s)
2023-01-11T22:37:44.3230524Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3230846Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79387
2023-01-11T22:37:44.3231064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79388
2023-01-11T22:37:44.3231437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3231616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3232002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3232193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3232543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3232718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3233158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3233345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3233589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3233834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3234285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3234690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3234919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3235127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3235246Z dist init r=0, world=2
2023-01-11T22:37:44.3235356Z dist init r=1, world=2
2023-01-11T22:37:44.3235457Z ok (3.911s)
2023-01-11T22:37:44.3235777Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3236096Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79466
2023-01-11T22:37:44.3236317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79467
2023-01-11T22:37:44.3236692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3236850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3237232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3237422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3237790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3237962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3238334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3238522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3238772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3239015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3239617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3240039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3240269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3240492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3240605Z dist init r=1, world=2
2023-01-11T22:37:44.3240717Z dist init r=0, world=2
2023-01-11T22:37:44.3240818Z ok (4.513s)
2023-01-11T22:37:44.3241142Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3241445Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79549
2023-01-11T22:37:44.3241663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79550
2023-01-11T22:37:44.3242120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3242296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3242674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3242864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3243302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3243485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3243860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3244030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3244274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3244519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3244920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3245314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3245547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3245776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3245888Z dist init r=0, world=2
2023-01-11T22:37:44.3245978Z dist init r=1, world=2
2023-01-11T22:37:44.3246080Z ok (4.512s)
2023-01-11T22:37:44.3246399Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3246719Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79632
2023-01-11T22:37:44.3246943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79633
2023-01-11T22:37:44.3247316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3247492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3247873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3248217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3248786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3248964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3249347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3249535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3249781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3250025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3250426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3250827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3251059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3251268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3251462Z dist init r=1, world=2
2023-01-11T22:37:44.3251574Z dist init r=0, world=2
2023-01-11T22:37:44.3251676Z ok (4.512s)
2023-01-11T22:37:44.3251995Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3252315Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79715
2023-01-11T22:37:44.3252577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79716
2023-01-11T22:37:44.3252964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3253120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3253501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3253696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3254065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3254238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3254612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3254806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3255052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3255276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3255677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3256081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3256309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3256537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3256647Z dist init r=0, world=2
2023-01-11T22:37:44.3256756Z dist init r=1, world=2
2023-01-11T22:37:44.3256857Z ok (4.513s)
2023-01-11T22:37:44.3257176Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3257592Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79798
2023-01-11T22:37:44.3257810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79799
2023-01-11T22:37:44.3258185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3258358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3258733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3258919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3259284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3259459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3259813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3260000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3260303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3260548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3260953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3261347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3261618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3261852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3262482Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3262603Z   warnings.warn(
2023-01-11T22:37:44.3263205Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3263318Z   warnings.warn(
2023-01-11T22:37:44.3263430Z dist init r=1, world=2
2023-01-11T22:37:44.3263538Z dist init r=0, world=2
2023-01-11T22:37:44.3263639Z ok (4.514s)
2023-01-11T22:37:44.3263956Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3264389Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79881
2023-01-11T22:37:44.3264611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79882
2023-01-11T22:37:44.3264967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3265143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3265519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3265708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3266068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3266245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3266621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3266810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3267038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3267289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3267686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3268076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3268304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3268532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3269559Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3269732Z   warnings.warn(
2023-01-11T22:37:44.3270780Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3270898Z   warnings.warn(
2023-01-11T22:37:44.3271519Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3271634Z   warnings.warn(
2023-01-11T22:37:44.3272236Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3272346Z   warnings.warn(
2023-01-11T22:37:44.3272457Z dist init r=1, world=2
2023-01-11T22:37:44.3272568Z dist init r=0, world=2
2023-01-11T22:37:44.3272670Z ok (4.413s)
2023-01-11T22:37:44.3272990Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3273421Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79964
2023-01-11T22:37:44.3273640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79965
2023-01-11T22:37:44.3274016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3274176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3274554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3274743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3275114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3275288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3275660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3275850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3276099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3276329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3276729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3277125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3277358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3277587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3277700Z dist init r=0, world=2
2023-01-11T22:37:44.3277811Z dist init r=1, world=2
2023-01-11T22:37:44.3277915Z ok (4.012s)
2023-01-11T22:37:44.3278211Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3278702Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80043
2023-01-11T22:37:44.3278920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80044
2023-01-11T22:37:44.3279290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3279804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3280288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3280490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3280867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3281043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3281403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3281591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3281835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3282081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3282486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3282885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3283116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3283343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3283461Z dist init r=1, world=2
2023-01-11T22:37:44.3283554Z dist init r=0, world=2
2023-01-11T22:37:44.3283655Z ok (3.912s)
2023-01-11T22:37:44.3283962Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3284391Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80122
2023-01-11T22:37:44.3284614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80123
2023-01-11T22:37:44.3284988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3285162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3285542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3285718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3286084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3286258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3286634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3286825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3287072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3287317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3287717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3288188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3288398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3288623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3288736Z dist init r=1, world=2
2023-01-11T22:37:44.3288846Z dist init r=0, world=2
2023-01-11T22:37:44.3288946Z ok (4.513s)
2023-01-11T22:37:44.3289295Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3289736Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80205
2023-01-11T22:37:44.3289955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80206
2023-01-11T22:37:44.3290314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3290490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3290868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3291058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3291426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3291597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3291969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3292160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3292387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3292636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3293035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3293434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3293667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3293894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3294921Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3295040Z   warnings.warn(
2023-01-11T22:37:44.3296050Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3296162Z   warnings.warn(
2023-01-11T22:37:44.3296276Z dist init r=1, world=2
2023-01-11T22:37:44.3296367Z dist init r=0, world=2
2023-01-11T22:37:44.3296472Z ok (4.412s)
2023-01-11T22:37:44.3296771Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3297267Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80288
2023-01-11T22:37:44.3297490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80289
2023-01-11T22:37:44.3297862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3298037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3298472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3298671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3299027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3299206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3299580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3299769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3300015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3300254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3300656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3301054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3301285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3301501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3301613Z dist init r=0, world=2
2023-01-11T22:37:44.3301727Z dist init r=1, world=2
2023-01-11T22:37:44.3301828Z ok (4.513s)
2023-01-11T22:37:44.3302129Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3302560Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80371
2023-01-11T22:37:44.3302786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80372
2023-01-11T22:37:44.3303161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3303318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3303695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3303889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3304250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3304422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3304839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3305032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3305277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3305501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3305905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3306367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3306596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3306828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3307894Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3308015Z   warnings.warn(
2023-01-11T22:37:44.3309025Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3309141Z   warnings.warn(
2023-01-11T22:37:44.3309254Z dist init r=0, world=2
2023-01-11T22:37:44.3309363Z dist init r=1, world=2
2023-01-11T22:37:44.3309451Z ok (4.513s)
2023-01-11T22:37:44.3309687Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3309999Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80454
2023-01-11T22:37:44.3310217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80455
2023-01-11T22:37:44.3310590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3310766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3311149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3311340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3311695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3311872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3312246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3312434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3312682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3312925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3313325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3313720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3313951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3314162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3314276Z dist init r=1, world=2
2023-01-11T22:37:44.3314387Z dist init r=0, world=2
2023-01-11T22:37:44.3314487Z ok (4.112s)
2023-01-11T22:37:44.3314724Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3315093Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80533
2023-01-11T22:37:44.3315311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80534
2023-01-11T22:37:44.3315685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3315842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3316263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3316460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3316824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3316999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3317374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3317564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3317810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3318056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3318440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3318835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3319061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3319290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3319645Z dist init r=1, world=2
2023-01-11T22:37:44.3319764Z dist init r=0, world=2
2023-01-11T22:37:44.3319867Z ok (4.111s)
2023-01-11T22:37:44.3320220Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80612
2023-01-11T22:37:44.3320422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80613
2023-01-11T22:37:44.3320808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3320986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3321364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3321558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3321931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3322105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3322480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3322670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3322898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3323144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3323546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3323947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3324261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3324489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3325575Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3325698Z   warnings.warn(
2023-01-11T22:37:44.3326710Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3326825Z   warnings.warn(
2023-01-11T22:37:44.3327063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3327281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3328040Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3328779Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3329023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3329250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3329364Z dist init r=1, world=2
2023-01-11T22:37:44.3329473Z dist init r=0, world=2
2023-01-11T22:37:44.3329574Z ok (4.511s)
2023-01-11T22:37:44.3329931Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80695
2023-01-11T22:37:44.3330150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80696
2023-01-11T22:37:44.3330511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3330692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3331072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3331268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3331630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3331807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3332187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3332375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3332603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3332848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3333330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3333724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3333954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3334225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3335259Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3335379Z   warnings.warn(
2023-01-11T22:37:44.3336386Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3336498Z   warnings.warn(
2023-01-11T22:37:44.3336736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3336953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3337709Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3338450Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3338689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3338923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3339038Z dist init r=1, world=2
2023-01-11T22:37:44.3339149Z dist init r=0, world=2
2023-01-11T22:37:44.3339250Z ok (4.511s)
2023-01-11T22:37:44.3339596Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80778
2023-01-11T22:37:44.3339819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80779
2023-01-11T22:37:44.3340175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3340352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3340734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3340926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3341291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3341466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3341838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3342081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3342325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3342551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3342956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3343398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3343628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3343846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3344863Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3344976Z   warnings.warn(
2023-01-11T22:37:44.3345974Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3346084Z   warnings.warn(
2023-01-11T22:37:44.3346318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3346549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3347284Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3348021Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:37:44.3348258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3348489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:44.3348600Z dist init r=0, world=2
2023-01-11T22:37:44.3348709Z dist init r=1, world=2
2023-01-11T22:37:44.3348808Z ok (4.511s)
2023-01-11T22:37:44.3349172Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80861
2023-01-11T22:37:44.3349388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80862
2023-01-11T22:37:44.3349759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3349917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3350291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3350538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3350901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3351071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3351441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3351630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3351921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3352155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3352560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3352950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3353179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3353404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3353515Z dist init r=0, world=2
2023-01-11T22:37:44.3353623Z dist init r=1, world=2
2023-01-11T22:37:44.3353720Z ok (4.514s)
2023-01-11T22:37:44.3354077Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80944
2023-01-11T22:37:44.3354292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80945
2023-01-11T22:37:44.3354659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3354829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3355208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3355399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3355757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3355926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3356300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3356471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3356714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3356958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3357352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3357740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3357967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3358194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3358813Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3358924Z   warnings.warn(
2023-01-11T22:37:44.3359767Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3359954Z   warnings.warn(
2023-01-11T22:37:44.3360057Z dist init r=1, world=2
2023-01-11T22:37:44.3360159Z dist init r=0, world=2
2023-01-11T22:37:44.3360252Z ok (4.513s)
2023-01-11T22:37:44.3360607Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81027
2023-01-11T22:37:44.3360919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81028
2023-01-11T22:37:44.3361310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3361486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3361846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3362037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3362402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3362574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3362947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3363131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3363379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3363621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3364005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3364404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3364630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3364847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3364951Z dist init r=1, world=2
2023-01-11T22:37:44.3365054Z dist init r=0, world=2
2023-01-11T22:37:44.3365149Z ok (4.413s)
2023-01-11T22:37:44.3365444Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81110
2023-01-11T22:37:44.3365644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81111
2023-01-11T22:37:44.3366015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3366179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3366559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3366747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3367102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3367272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3367644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3367823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3368052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3368290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3368749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3369136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3369355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3369575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3369679Z dist init r=1, world=2
2023-01-11T22:37:44.3369826Z dist init r=0, world=2
2023-01-11T22:37:44.3369915Z ok (4.010s)
2023-01-11T22:37:44.3370320Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81189
2023-01-11T22:37:44.3370527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81190
2023-01-11T22:37:44.3370900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3371068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3371440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3371621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3371979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3372150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3372505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3372685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3372926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3373166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3373561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3373943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3374168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3374387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3375408Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3375521Z   warnings.warn(
2023-01-11T22:37:44.3376521Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3376616Z   warnings.warn(
2023-01-11T22:37:44.3377230Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3377394Z   warnings.warn(
2023-01-11T22:37:44.3378009Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3378115Z   warnings.warn(
2023-01-11T22:37:44.3378218Z dist init r=0, world=2
2023-01-11T22:37:44.3378320Z dist init r=1, world=2
2023-01-11T22:37:44.3378412Z ok (3.913s)
2023-01-11T22:37:44.3378850Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81268
2023-01-11T22:37:44.3379062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81269
2023-01-11T22:37:44.3379428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3379606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3379974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3380160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3380521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3380694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3381061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3381230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3381467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3381704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3382105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3382490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3382710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3382934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3383649Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3383755Z   warnings.warn(
2023-01-11T22:37:44.3384469Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3384563Z   warnings.warn(
2023-01-11T22:37:44.3384675Z dist init r=0, world=2
2023-01-11T22:37:44.3384777Z dist init r=1, world=2
2023-01-11T22:37:44.3384872Z ok (3.912s)
2023-01-11T22:37:44.3385268Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81347
2023-01-11T22:37:44.3385479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81348
2023-01-11T22:37:44.3385848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3386072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3386437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3386623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3386980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3387151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3387564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3387803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3388044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3388280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3388686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3389064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3389284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3389500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3390521Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3390636Z   warnings.warn(
2023-01-11T22:37:44.3391640Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3391745Z   warnings.warn(
2023-01-11T22:37:44.3392360Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3392464Z   warnings.warn(
2023-01-11T22:37:44.3393073Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:37:44.3393178Z   warnings.warn(
2023-01-11T22:37:44.3393284Z dist init r=1, world=2
2023-01-11T22:37:44.3393377Z dist init r=0, world=2
2023-01-11T22:37:44.3393475Z ok (4.012s)
2023-01-11T22:37:44.3393874Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81426
2023-01-11T22:37:44.3394086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81427
2023-01-11T22:37:44.3394446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3394616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3394986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3395237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3395584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3395748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3396118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3396349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3396594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3396834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3397233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3397627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3397851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3398063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3398776Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3398886Z   warnings.warn(
2023-01-11T22:37:44.3399835Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3399952Z   warnings.warn(
2023-01-11T22:37:44.3400068Z dist init r=1, world=2
2023-01-11T22:37:44.3400172Z dist init r=0, world=2
2023-01-11T22:37:44.3400263Z ok (4.012s)
2023-01-11T22:37:44.3400657Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81505
2023-01-11T22:37:44.3400863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81506
2023-01-11T22:37:44.3401236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3401406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3401778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3401962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3402318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3402479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3402848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3403036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3403265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3403496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3403887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3404369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3404589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3404847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3405921Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3406037Z   warnings.warn(
2023-01-11T22:37:44.3407044Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3407155Z   warnings.warn(
2023-01-11T22:37:44.3407258Z dist init r=0, world=2
2023-01-11T22:37:44.3407351Z dist init r=1, world=2
2023-01-11T22:37:44.3407449Z ok (4.012s)
2023-01-11T22:37:44.3407838Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81584
2023-01-11T22:37:44.3408056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81585
2023-01-11T22:37:44.3408421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3408597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3408971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3409159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3409504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3409675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3410042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3410218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3410460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3410704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3411095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3411488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3411719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3411932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3412645Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3412817Z   warnings.warn(
2023-01-11T22:37:44.3413524Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3413629Z   warnings.warn(
2023-01-11T22:37:44.3413741Z dist init r=1, world=2
2023-01-11T22:37:44.3413842Z dist init r=0, world=2
2023-01-11T22:37:44.3413939Z ok (4.012s)
2023-01-11T22:37:44.3414371Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81663
2023-01-11T22:37:44.3414584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81664
2023-01-11T22:37:44.3414946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3415118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3415488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3415672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3416031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3416197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3416565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3416744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3416974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3417215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3417621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3418010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3418233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3418457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3419475Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3419586Z   warnings.warn(
2023-01-11T22:37:44.3420583Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3420688Z   warnings.warn(
2023-01-11T22:37:44.3420791Z dist init r=1, world=2
2023-01-11T22:37:44.3420883Z dist init r=0, world=2
2023-01-11T22:37:44.3420977Z ok (4.012s)
2023-01-11T22:37:44.3421357Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81742
2023-01-11T22:37:44.3421629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81743
2023-01-11T22:37:44.3421992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3422161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3422537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3422775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3423132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3423300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3423662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3423848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3424085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3424325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3424723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3425117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3425342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3425553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3426266Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3426375Z   warnings.warn(
2023-01-11T22:37:44.3427087Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:37:44.3427200Z   warnings.warn(
2023-01-11T22:37:44.3427306Z dist init r=0, world=2
2023-01-11T22:37:44.3427411Z dist init r=1, world=2
2023-01-11T22:37:44.3427503Z ok (4.012s)
2023-01-11T22:37:44.3427786Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3428088Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81821
2023-01-11T22:37:44.3428303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81822
2023-01-11T22:37:44.3428667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3428832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3429203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3429393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3429744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3429911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3430266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3430510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3430753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3430988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3431385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3431816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3432046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3432265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3432374Z dist init r=1, world=2
2023-01-11T22:37:44.3432470Z dist init r=0, world=2
2023-01-11T22:37:44.3432560Z ok (4.212s)
2023-01-11T22:37:44.3432840Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3433157Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81900
2023-01-11T22:37:44.3433368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81901
2023-01-11T22:37:44.3433742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3433908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3434285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3434457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3434818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3434987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3435358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3435537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3435782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3436018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3436409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3436801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3437016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3437242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3437351Z dist init r=0, world=2
2023-01-11T22:37:44.3437457Z dist init r=1, world=2
2023-01-11T22:37:44.3437556Z ok (3.911s)
2023-01-11T22:37:44.3437832Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3438151Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81979
2023-01-11T22:37:44.3438367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81980
2023-01-11T22:37:44.3438725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3438953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3439334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3439751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3440126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3440297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3440738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3440934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3441179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3441403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3441806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3442193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3442422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3442650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3442766Z dist init r=0, world=2
2023-01-11T22:37:44.3442874Z dist init r=1, world=2
2023-01-11T22:37:44.3442974Z ok (4.112s)
2023-01-11T22:37:44.3443234Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:37:44.3443546Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82058
2023-01-11T22:37:44.3443765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82059
2023-01-11T22:37:44.3444137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3444307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3444682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3444876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3445236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3445410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3445768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3445955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3446198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3446436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3446833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3447230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3447457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3447681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3447776Z dist init r=1, world=2
2023-01-11T22:37:44.3447882Z dist init r=0, world=2
2023-01-11T22:37:44.3448051Z ok (4.112s)
2023-01-11T22:37:44.3448412Z   test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82137
2023-01-11T22:37:44.3448631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82138
2023-01-11T22:37:44.3449003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3449218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3449604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3449789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3450132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3450302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3450667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3450853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3451100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3451337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3451731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3452119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3452331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3452561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3452670Z dist init r=0, world=2
2023-01-11T22:37:44.3452779Z dist init r=1, world=2
2023-01-11T22:37:44.3452875Z ok (4.111s)
2023-01-11T22:37:44.3453239Z   test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82216
2023-01-11T22:37:44.3453453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82217
2023-01-11T22:37:44.3453827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3453984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3454361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3454548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3454914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3455086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3455450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3455635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3455878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3456117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3456498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3456884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3457178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3457407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3457517Z dist init r=1, world=2
2023-01-11T22:37:44.3457625Z dist init r=0, world=2
2023-01-11T22:37:44.3457720Z ok (4.111s)
2023-01-11T22:37:44.3458120Z   test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82295
2023-01-11T22:37:44.3458328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82296
2023-01-11T22:37:44.3458705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3458877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3459260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3459450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3459810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3459982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3460357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3460547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3460774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3461017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3461413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3461810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3462032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3462257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3462366Z dist init r=1, world=2
2023-01-11T22:37:44.3462474Z dist init r=0, world=2
2023-01-11T22:37:44.3462557Z ok (4.011s)
2023-01-11T22:37:44.3462864Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82374
2023-01-11T22:37:44.3463078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82375
2023-01-11T22:37:44.3463444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3463623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3463997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3464186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3464549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:37:44.3464706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:44.3465082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:44.3465268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:44.3465509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:37:44.3465804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:37:44.3466200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3466590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:37:44.3466812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:44.3467079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:37:44.3468120Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3468236Z   warnings.warn(
2023-01-11T22:37:44.3469226Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:37:44.3469335Z   warnings.warn(
2023-01-11T22:37:44.3469447Z dist init r=0, world=2
2023-01-11T22:37:44.3469555Z dist init r=1, world=2
2023-01-11T22:37:44.3469653Z ok (4.010s)
2023-01-11T22:37:44.3469676Z 
2023-01-11T22:37:44.3469942Z ----------------------------------------------------------------------
2023-01-11T22:37:44.3470061Z Ran 116 tests in 476.903s
2023-01-11T22:37:44.3470082Z 
2023-01-11T22:37:44.3470174Z OK
2023-01-11T22:37:44.3470193Z 
2023-01-11T22:37:44.3470313Z Generating XML reports...
2023-01-11T22:37:44.3470747Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20230111222946.xml
2023-01-11T22:37:44.3470767Z 
2023-01-11T22:37:44.3471138Z ##[endgroup]
2023-01-11T22:37:44.3471616Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_2awsfp6_)
2023-01-11T22:37:44.3471641Z 
2023-01-11T22:37:44.3471889Z Running distributed/test_c10d_nccl ... [2023-01-11 22:37:44.251321]
2023-01-11T22:37:44.3472379Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:37:44.251692]
2023-01-11T22:55:24.2969879Z 
2023-01-11T22:55:24.2970586Z Expand the folded group to see the log file of distributed/test_c10d_nccl
2023-01-11T22:55:24.2972998Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_fkrwsvpj)
2023-01-11T22:55:24.2976564Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2023-01-11T22:55:24.2979006Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2023-01-11T22:55:24.2979371Z test_broadcast_coalesced_nccl (__main__.CommTest)
2023-01-11T22:55:24.2979660Z test_nccl_barrier (__main__.CommTest)
2023-01-11T22:55:24.2979990Z test_nccl_barrier_device_ids (__main__.CommTest)
2023-01-11T22:55:24.2980341Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2023-01-11T22:55:24.2981253Z test_nccl_barrier_timeout (__main__.CommTest)
2023-01-11T22:55:24.2981608Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2023-01-11T22:55:24.2982170Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2023-01-11T22:55:24.2982550Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2023-01-11T22:55:24.2984204Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2023-01-11T22:55:24.2984943Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2023-01-11T22:55:24.2985629Z test_nncl_rank_membership (__main__.CommTest)
2023-01-11T22:55:24.2986046Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2023-01-11T22:55:24.2986440Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2023-01-11T22:55:24.2987022Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2023-01-11T22:55:24.2987411Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2023-01-11T22:55:24.2987788Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2023-01-11T22:55:24.2988291Z test_tensor_dtype_complex (__main__.CommTest)
2023-01-11T22:55:24.2988708Z test_tensor_dtype_mismatch (__main__.CommTest)
2023-01-11T22:55:24.2990165Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_into_tensor_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_alltoall_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_tensor_work_wait_gpu>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2023-01-11T22:55:24.2991663Z test_allgather_into_tensor_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2992218Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2992633Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2992970Z test_alltoall_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2993415Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2993942Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2994308Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2023-01-11T22:55:24.2994828Z test_reduce_scatter_tensor_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2995277Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.2995621Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:55:24.3007530Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2023-01-11T22:55:24.3018507Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3019013Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3019624Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3020206Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3020658Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3021336Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3021779Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3022397Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3022885Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3023344Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3024032Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3024507Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3025221Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3025743Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3026514Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3027949Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3028418Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3029123Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3029656Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3030350Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3030908Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3031545Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3032183Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3032701Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3033401Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3033870Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3034496Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3035096Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3035518Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3035924Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3036341Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3036872Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3037456Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3037869Z test_failure_recovery (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3038324Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3038796Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3039247Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3040402Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3040944Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3041652Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3042082Z test_fp16 (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3042495Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3042926Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3043327Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3043787Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3044250Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3044688Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3045110Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3045591Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3046075Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3046572Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3047027Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3047464Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3047905Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3048631Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3049133Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3049634Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3050113Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3050474Z test_no_grad (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3050911Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3051430Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3051836Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3052290Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3052753Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3053183Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3053619Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:55:24.3054893Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2023-01-11T22:55:24.3056129Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3056537Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3056957Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3057346Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3057754Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3058160Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3058534Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3058954Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3059313Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2023-01-11T22:55:24.3060525Z <unittest.suite.TestSuite tests=[<__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_all_to_all_single>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_base>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]>
2023-01-11T22:55:24.3061762Z test_all_to_all_single (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:55:24.3062528Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:55:24.3063077Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:55:24.3063614Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:55:24.3064110Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:55:24.3064674Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2023-01-11T22:55:24.3065150Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2023-01-11T22:55:24.3067240Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2023-01-11T22:55:24.3069351Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3069725Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3070098Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3070457Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3070807Z test_barrier (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3071144Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3071499Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3071849Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3072191Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3072538Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3072920Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3073261Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3073655Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3074030Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3074421Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3074760Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3075139Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3075496Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3075826Z test_send_recv (__main__.ProcessGroupNCCLTest)
2023-01-11T22:55:24.3076231Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2023-01-11T22:55:24.3076638Z test_common_errors (__main__.RendezvousEnvTest)
2023-01-11T22:55:24.3076944Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:55:24.3077362Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2023-01-11T22:55:24.3077785Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2023-01-11T22:55:24.3078483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3078920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3080107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3080600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3080830Z 
2023-01-11T22:55:24.3080923Z Running tests...
2023-01-11T22:55:24.3081340Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3081873Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3082347Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3082792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82521
2023-01-11T22:55:24.3083238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82522
2023-01-11T22:55:24.3083850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3084283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3084963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3085428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3086002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3086423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3087057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3087529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3087947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3088625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3089166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3089655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3090304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3090990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3092037Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:55:24.3092662Z   warnings.warn(
2023-01-11T22:55:24.3093536Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:55:24.3094181Z   warnings.warn(
2023-01-11T22:55:24.3094418Z ok (6.615s)
2023-01-11T22:55:24.3094564Z 
2023-01-11T22:55:24.3094836Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3095148Z Ran 1 test in 6.615s
2023-01-11T22:55:24.3095308Z 
2023-01-11T22:55:24.3095403Z OK
2023-01-11T22:55:24.3095536Z 
2023-01-11T22:55:24.3095664Z Generating XML reports...
2023-01-11T22:55:24.3096211Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223748.xml
2023-01-11T22:55:24.3096853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3097299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3097870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3098322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3098551Z 
2023-01-11T22:55:24.3098661Z Running tests...
2023-01-11T22:55:24.3099061Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3099587Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3100049Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3100511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82638
2023-01-11T22:55:24.3100959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82639
2023-01-11T22:55:24.3101546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3102093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3102668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3103132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3103690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3104132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3104746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3105216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3105635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3106126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3106618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3107080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3107739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3108424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3108817Z ok (6.633s)
2023-01-11T22:55:24.3108947Z 
2023-01-11T22:55:24.3109212Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3109539Z Ran 1 test in 6.633s
2023-01-11T22:55:24.3109699Z 
2023-01-11T22:55:24.3109793Z OK
2023-01-11T22:55:24.3109927Z 
2023-01-11T22:55:24.3110033Z Generating XML reports...
2023-01-11T22:55:24.3110573Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223757.xml
2023-01-11T22:55:24.3111230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3111675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3112235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3112697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3112925Z 
2023-01-11T22:55:24.3113039Z Running tests...
2023-01-11T22:55:24.3113425Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3113951Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3114408Z   test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3114852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82755
2023-01-11T22:55:24.3115284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82756
2023-01-11T22:55:24.3115880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3116326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3116876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3117349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3117925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3118366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3118918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3120016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3120540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3121010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3121384Z skip: Need at least 4 CUDA devices (3.918s)
2023-01-11T22:55:24.3121575Z 
2023-01-11T22:55:24.3121856Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3122179Z Ran 1 test in 3.918s
2023-01-11T22:55:24.3122431Z 
2023-01-11T22:55:24.3122533Z OK (skipped=1)
2023-01-11T22:55:24.3122687Z 
2023-01-11T22:55:24.3122812Z Generating XML reports...
2023-01-11T22:55:24.3123357Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223805.xml
2023-01-11T22:55:24.3123993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3124444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3125016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3125480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3125709Z 
2023-01-11T22:55:24.3125802Z Running tests...
2023-01-11T22:55:24.3126194Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3126722Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3127191Z   test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3127632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82858
2023-01-11T22:55:24.3128075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82859
2023-01-11T22:55:24.3128685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3129112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3129680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3130144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3130718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3131144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3131710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3132174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3132592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3133087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3133573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3134054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3134690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3135377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3135768Z ok (6.612s)
2023-01-11T22:55:24.3135919Z 
2023-01-11T22:55:24.3136186Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3136493Z Ran 1 test in 6.612s
2023-01-11T22:55:24.3136652Z 
2023-01-11T22:55:24.3136746Z OK
2023-01-11T22:55:24.3136963Z 
2023-01-11T22:55:24.3137088Z Generating XML reports...
2023-01-11T22:55:24.3137615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223812.xml
2023-01-11T22:55:24.3138273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3138717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3139293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3139788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3140021Z 
2023-01-11T22:55:24.3140131Z Running tests...
2023-01-11T22:55:24.3140754Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3141328Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3141953Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3142535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82975
2023-01-11T22:55:24.3142984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82976
2023-01-11T22:55:24.3143578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3144024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3144595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3145043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3145622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3146065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3146640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3147084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3147518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3148004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3148472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3148954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3149614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3150298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3150676Z ok (3.920s)
2023-01-11T22:55:24.3150824Z 
2023-01-11T22:55:24.3151091Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3151416Z Ran 1 test in 3.920s
2023-01-11T22:55:24.3151576Z 
2023-01-11T22:55:24.3151670Z OK
2023-01-11T22:55:24.3151785Z 
2023-01-11T22:55:24.3151907Z Generating XML reports...
2023-01-11T22:55:24.3152448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223821.xml
2023-01-11T22:55:24.3153112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3153540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3154111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3154573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3154879Z 
2023-01-11T22:55:24.3154991Z Running tests...
2023-01-11T22:55:24.3155381Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3155906Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3156373Z   test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3156814Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83082
2023-01-11T22:55:24.3157310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83083
2023-01-11T22:55:24.3157923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3158370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3158923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3160047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3161064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3161813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3162710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3163183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3163620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3164072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3164457Z skip: Need at least 4 CUDA devices (3.916s)
2023-01-11T22:55:24.3164648Z 
2023-01-11T22:55:24.3164921Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3165236Z Ran 1 test in 3.917s
2023-01-11T22:55:24.3165397Z 
2023-01-11T22:55:24.3165507Z OK (skipped=1)
2023-01-11T22:55:24.3165662Z 
2023-01-11T22:55:24.3165785Z Generating XML reports...
2023-01-11T22:55:24.3166321Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223827.xml
2023-01-11T22:55:24.3166965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3167415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3167988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3168435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3168662Z 
2023-01-11T22:55:24.3168771Z Running tests...
2023-01-11T22:55:24.3169178Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3169705Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3170167Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3170636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83185
2023-01-11T22:55:24.3171082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83186
2023-01-11T22:55:24.3171671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3172118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3172685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3173147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3173705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3174257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3174829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3175289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3175706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3176236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3176634Z skip: Need at least 4 CUDA devices (3.897s)
2023-01-11T22:55:24.3176822Z 
2023-01-11T22:55:24.3177076Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3177399Z Ran 1 test in 3.897s
2023-01-11T22:55:24.3177559Z 
2023-01-11T22:55:24.3177666Z OK (skipped=1)
2023-01-11T22:55:24.3177824Z 
2023-01-11T22:55:24.3177949Z Generating XML reports...
2023-01-11T22:55:24.3178474Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223833.xml
2023-01-11T22:55:24.3179129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3179576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3180130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3180602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3180830Z 
2023-01-11T22:55:24.3180940Z Running tests...
2023-01-11T22:55:24.3181340Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3181847Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3182347Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3182830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83288
2023-01-11T22:55:24.3183254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83289
2023-01-11T22:55:24.3183853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3184301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3184880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3185322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3185895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3186334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3186908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3187348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3187783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3188251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3188624Z skip: Need at least 4 CUDA devices (3.902s)
2023-01-11T22:55:24.3188818Z 
2023-01-11T22:55:24.3189088Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3189411Z Ran 1 test in 3.903s
2023-01-11T22:55:24.3189571Z 
2023-01-11T22:55:24.3189677Z OK (skipped=1)
2023-01-11T22:55:24.3189813Z 
2023-01-11T22:55:24.3189937Z Generating XML reports...
2023-01-11T22:55:24.3190474Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223839.xml
2023-01-11T22:55:24.3191255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3191684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3192252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3192715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3192941Z 
2023-01-11T22:55:24.3193052Z Running tests...
2023-01-11T22:55:24.3193483Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3194064Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3194554Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3195008Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83391
2023-01-11T22:55:24.3195453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83392
2023-01-11T22:55:24.3196056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3196501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3197052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3197520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3198094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3198517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3199084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3199997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3200438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3200889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3201371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3201865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3202542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3203214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3203747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:55:24.3204238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:55:24.3204872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3205546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3205929Z ok (6.612s)
2023-01-11T22:55:24.3206078Z 
2023-01-11T22:55:24.3206350Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3206657Z Ran 1 test in 6.612s
2023-01-11T22:55:24.3206816Z 
2023-01-11T22:55:24.3206909Z OK
2023-01-11T22:55:24.3207044Z 
2023-01-11T22:55:24.3207165Z Generating XML reports...
2023-01-11T22:55:24.3207692Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223846.xml
2023-01-11T22:55:24.3208356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3208897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3209475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3209929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3210155Z 
2023-01-11T22:55:24.3210265Z Running tests...
2023-01-11T22:55:24.3210664Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3211233Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3211731Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3212197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83523
2023-01-11T22:55:24.3212639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83524
2023-01-11T22:55:24.3213231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3213674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3214246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3214716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3215274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3215713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3216282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3216723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3217160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3217646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3218131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3218588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3219249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3219782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:55:24.3220428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3220935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:55:24.3221579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3222258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3222631Z ok (6.618s)
2023-01-11T22:55:24.3222780Z 
2023-01-11T22:55:24.3223040Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3223364Z Ran 1 test in 6.618s
2023-01-11T22:55:24.3223524Z 
2023-01-11T22:55:24.3223622Z OK
2023-01-11T22:55:24.3223737Z 
2023-01-11T22:55:24.3223859Z Generating XML reports...
2023-01-11T22:55:24.3224399Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223854.xml
2023-01-11T22:55:24.3225050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3225475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3226144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3226612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3226838Z 
2023-01-11T22:55:24.3226949Z Running tests...
2023-01-11T22:55:24.3227333Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3227854Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3228388Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3228846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83646
2023-01-11T22:55:24.3229291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83647
2023-01-11T22:55:24.3229888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3230337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3230891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3231352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3231927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3232372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3232925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3233387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3233820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3234294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3234773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3235261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3235916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3236437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:55:24.3237078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3237603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:55:24.3238247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3238915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3239308Z ok (6.632s)
2023-01-11T22:55:24.3239822Z 
2023-01-11T22:55:24.3240130Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3240440Z Ran 1 test in 6.632s
2023-01-11T22:55:24.3240601Z 
2023-01-11T22:55:24.3240694Z OK
2023-01-11T22:55:24.3240825Z 
2023-01-11T22:55:24.3240948Z Generating XML reports...
2023-01-11T22:55:24.3241492Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223903.xml
2023-01-11T22:55:24.3242133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3242580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3243150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3243707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3243934Z 
2023-01-11T22:55:24.3244043Z Running tests...
2023-01-11T22:55:24.3244447Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3244973Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3245423Z   test_nncl_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3245939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83769
2023-01-11T22:55:24.3246396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83770
2023-01-11T22:55:24.3246985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3247431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3248006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3248467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3249021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3249464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3250038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3250483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3250916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3251403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3251890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3252352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3253004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3253535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:55:24.3254186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3254695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:55:24.3255336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3256009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3256401Z ok (3.937s)
2023-01-11T22:55:24.3256531Z 
2023-01-11T22:55:24.3256797Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3257120Z Ran 1 test in 3.938s
2023-01-11T22:55:24.3257279Z 
2023-01-11T22:55:24.3257378Z OK
2023-01-11T22:55:24.3257493Z 
2023-01-11T22:55:24.3257619Z Generating XML reports...
2023-01-11T22:55:24.3258157Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223912.xml
2023-01-11T22:55:24.3258821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3259250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3259822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3260285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3260601Z 
2023-01-11T22:55:24.3260715Z Running tests...
2023-01-11T22:55:24.3261105Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3261627Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3262126Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3262609Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83878
2023-01-11T22:55:24.3263102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83879
2023-01-11T22:55:24.3263716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3264158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3264709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3265179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3265757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3266195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3266743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3267206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3267642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3268112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3268591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3269082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3269736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3270245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:55:24.3270890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3271412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:55:24.3272055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3272718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3273112Z ok (6.636s)
2023-01-11T22:55:24.3273257Z 
2023-01-11T22:55:24.3273519Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3273824Z Ran 1 test in 6.636s
2023-01-11T22:55:24.3273985Z 
2023-01-11T22:55:24.3274078Z OK
2023-01-11T22:55:24.3274213Z 
2023-01-11T22:55:24.3274337Z Generating XML reports...
2023-01-11T22:55:24.3274877Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223919.xml
2023-01-11T22:55:24.3275522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3275966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3276536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3276984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3277212Z 
2023-01-11T22:55:24.3277389Z Running tests...
2023-01-11T22:55:24.3277790Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3278310Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3278788Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3279264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83999
2023-01-11T22:55:24.3280118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84000
2023-01-11T22:55:24.3280801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3281256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3281834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3282304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3282863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3283353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3283924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3284380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3284799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3285286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3285771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3286228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3286884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3287565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3288090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:55:24.3288560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:55:24.3289213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3289892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3290286Z ok (5.621s)
2023-01-11T22:55:24.3290416Z 
2023-01-11T22:55:24.3290685Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3291012Z Ran 1 test in 5.621s
2023-01-11T22:55:24.3291176Z 
2023-01-11T22:55:24.3291272Z OK
2023-01-11T22:55:24.3291407Z 
2023-01-11T22:55:24.3291514Z Generating XML reports...
2023-01-11T22:55:24.3292055Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223928.xml
2023-01-11T22:55:24.3292713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3293166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3293720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3294235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3294462Z 
2023-01-11T22:55:24.3294573Z Running tests...
2023-01-11T22:55:24.3294957Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3295573Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3296068Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3296546Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84121
2023-01-11T22:55:24.3296972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84122
2023-01-11T22:55:24.3297620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3298073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3298630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3299099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3299679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3300125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3300677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3301139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3301572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3302050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3302421Z skip: Need at least 4 CUDA devices (3.955s)
2023-01-11T22:55:24.3302617Z 
2023-01-11T22:55:24.3302886Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3303211Z Ran 1 test in 3.955s
2023-01-11T22:55:24.3303372Z 
2023-01-11T22:55:24.3303463Z OK (skipped=1)
2023-01-11T22:55:24.3303617Z 
2023-01-11T22:55:24.3303743Z Generating XML reports...
2023-01-11T22:55:24.3304277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223936.xml
2023-01-11T22:55:24.3304929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3305357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3305928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3306393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3306619Z 
2023-01-11T22:55:24.3306711Z Running tests...
2023-01-11T22:55:24.3307109Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3307633Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3308126Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3308579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84224
2023-01-11T22:55:24.3309026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84225
2023-01-11T22:55:24.3309629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3310054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3310611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3311057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3311627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3312078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3312728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3313188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3313608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3314079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3314605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3315101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3315742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3316426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3316821Z ok (5.525s)
2023-01-11T22:55:24.3316970Z 
2023-01-11T22:55:24.3317234Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3317541Z Ran 1 test in 5.525s
2023-01-11T22:55:24.3317701Z 
2023-01-11T22:55:24.3317796Z OK
2023-01-11T22:55:24.3317929Z 
2023-01-11T22:55:24.3318053Z Generating XML reports...
2023-01-11T22:55:24.3318572Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223942.xml
2023-01-11T22:55:24.3319235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3320094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3320683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3321133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3321370Z 
2023-01-11T22:55:24.3321480Z Running tests...
2023-01-11T22:55:24.3321879Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3322388Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3322874Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3323342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84340
2023-01-11T22:55:24.3323790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84341
2023-01-11T22:55:24.3324378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3324824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3325391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3325845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3326417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3326856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3327428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3327875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3328312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3328796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3329259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3329739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3330497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3331022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:55:24.3331651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3332243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:55:24.3332893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3333569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:55:24.3333937Z ok (5.532s)
2023-01-11T22:55:24.3334090Z 
2023-01-11T22:55:24.3334354Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3334679Z Ran 1 test in 5.532s
2023-01-11T22:55:24.3334839Z 
2023-01-11T22:55:24.3334915Z OK
2023-01-11T22:55:24.3335046Z 
2023-01-11T22:55:24.3335171Z Generating XML reports...
2023-01-11T22:55:24.3335709Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223950.xml
2023-01-11T22:55:24.3336371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3336805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3337376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3337840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3338071Z 
2023-01-11T22:55:24.3338180Z Running tests...
2023-01-11T22:55:24.3338565Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3339091Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3339559Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3339996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84460
2023-01-11T22:55:24.3340442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84461
2023-01-11T22:55:24.3341049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3341495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3342032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3342470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3343043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3343490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3344070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3344528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3344965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3345436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3345911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3346394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3347022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3347792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3348188Z ok (6.609s)
2023-01-11T22:55:24.3348339Z 
2023-01-11T22:55:24.3348609Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3348918Z Ran 1 test in 6.609s
2023-01-11T22:55:24.3349078Z 
2023-01-11T22:55:24.3349172Z OK
2023-01-11T22:55:24.3349305Z 
2023-01-11T22:55:24.3349476Z Generating XML reports...
2023-01-11T22:55:24.3350007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223957.xml
2023-01-11T22:55:24.3350661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3351106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3351682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3352127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3352354Z 
2023-01-11T22:55:24.3352462Z Running tests...
2023-01-11T22:55:24.3352863Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3353370Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3353843Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3354301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84577
2023-01-11T22:55:24.3354743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84578
2023-01-11T22:55:24.3355328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3355778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3356348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3356814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3357368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3357808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3358378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3358821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3359255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3360191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3360687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3361148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3361821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3362503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3363554Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:55:24.3364174Z   warnings.warn(
2023-01-11T22:55:24.3365043Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:55:24.3365763Z   warnings.warn(
2023-01-11T22:55:24.3366690Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:55:24.3367284Z   warnings.warn(
2023-01-11T22:55:24.3368147Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:55:24.3368753Z   warnings.warn(
2023-01-11T22:55:24.3368996Z ok (5.427s)
2023-01-11T22:55:24.3369127Z 
2023-01-11T22:55:24.3369398Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3369724Z Ran 1 test in 5.428s
2023-01-11T22:55:24.3369885Z 
2023-01-11T22:55:24.3369977Z OK
2023-01-11T22:55:24.3370111Z 
2023-01-11T22:55:24.3370219Z Generating XML reports...
2023-01-11T22:55:24.3370761Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224006.xml
2023-01-11T22:55:24.3371425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3371870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3372424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3372884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3373116Z 
2023-01-11T22:55:24.3373225Z Running tests...
2023-01-11T22:55:24.3373609Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3374130Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3374631Z   test_allgather_into_tensor_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3375110Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84686
2023-01-11T22:55:24.3375544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84687
2023-01-11T22:55:24.3376151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3376597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3377168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3377624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3378205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3378645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3379197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3379658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3380096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3380568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3381033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3381525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3382246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3382931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3383885Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3384611Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3385459Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3386173Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3386478Z ok (6.738s)
2023-01-11T22:55:24.3386625Z 
2023-01-11T22:55:24.3386888Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3387214Z Ran 1 test in 6.738s
2023-01-11T22:55:24.3387373Z 
2023-01-11T22:55:24.3387449Z OK
2023-01-11T22:55:24.3387582Z 
2023-01-11T22:55:24.3387705Z Generating XML reports...
2023-01-11T22:55:24.3388263Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224014.xml
2023-01-11T22:55:24.3388940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3389367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3389934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3390403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3390631Z 
2023-01-11T22:55:24.3390739Z Running tests...
2023-01-11T22:55:24.3391124Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3391642Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3392122Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3392574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84801
2023-01-11T22:55:24.3393015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84802
2023-01-11T22:55:24.3393615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3394091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3394668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3395136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3395712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3396132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3396699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3397158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3397590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3398038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3398512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3399076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3400091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3400796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3401808Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3402535Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3403388Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3404078Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3404407Z ok (6.721s)
2023-01-11T22:55:24.3404555Z 
2023-01-11T22:55:24.3404819Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3405123Z Ran 1 test in 6.722s
2023-01-11T22:55:24.3405281Z 
2023-01-11T22:55:24.3405375Z OK
2023-01-11T22:55:24.3405509Z 
2023-01-11T22:55:24.3405634Z Generating XML reports...
2023-01-11T22:55:24.3406196Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224023.xml
2023-01-11T22:55:24.3406855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3407295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3407870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3408353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3408556Z 
2023-01-11T22:55:24.3408666Z Running tests...
2023-01-11T22:55:24.3409060Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3409583Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3410052Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3410515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84916
2023-01-11T22:55:24.3410955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84917
2023-01-11T22:55:24.3411543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3411998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3412574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3413037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3413593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3414033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3414605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3415063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3415481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3415951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3416524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3416998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3417662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3418350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3419321Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3420026Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3420871Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3421576Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3421895Z ok (6.711s)
2023-01-11T22:55:24.3422044Z 
2023-01-11T22:55:24.3422295Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3422619Z Ran 1 test in 6.711s
2023-01-11T22:55:24.3422782Z 
2023-01-11T22:55:24.3422877Z OK
2023-01-11T22:55:24.3423010Z 
2023-01-11T22:55:24.3423133Z Generating XML reports...
2023-01-11T22:55:24.3423673Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224032.xml
2023-01-11T22:55:24.3424341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3424788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3425341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3425808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3426036Z 
2023-01-11T22:55:24.3426143Z Running tests...
2023-01-11T22:55:24.3426542Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3427053Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3427530Z   test_alltoall_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3427996Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85031
2023-01-11T22:55:24.3428423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85032
2023-01-11T22:55:24.3429022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3429474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3430040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3430485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3431062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3431505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3432058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3432519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3432949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3433479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3433951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3434439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3435096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3435828Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3436750Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3437461Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3438316Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3439020Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3439326Z ok (6.720s)
2023-01-11T22:55:24.3439807Z 
2023-01-11T22:55:24.3440189Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3440507Z Ran 1 test in 6.721s
2023-01-11T22:55:24.3440669Z 
2023-01-11T22:55:24.3440762Z OK
2023-01-11T22:55:24.3440877Z 
2023-01-11T22:55:24.3441005Z Generating XML reports...
2023-01-11T22:55:24.3441561Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224041.xml
2023-01-11T22:55:24.3442235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3442670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3443245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3443712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3443941Z 
2023-01-11T22:55:24.3444049Z Running tests...
2023-01-11T22:55:24.3444437Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3444963Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3445448Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3445899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85148
2023-01-11T22:55:24.3446345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85149
2023-01-11T22:55:24.3446954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3447399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3447953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3448413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3448993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3449415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3449991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3450451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3450991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3451442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3451920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3452413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3453132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3453816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3454736Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3455450Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3456291Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3456971Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3457302Z ok (6.673s)
2023-01-11T22:55:24.3457448Z 
2023-01-11T22:55:24.3457712Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3458035Z Ran 1 test in 6.673s
2023-01-11T22:55:24.3458179Z 
2023-01-11T22:55:24.3458274Z OK
2023-01-11T22:55:24.3458404Z 
2023-01-11T22:55:24.3458526Z Generating XML reports...
2023-01-11T22:55:24.3459078Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224050.xml
2023-01-11T22:55:24.3459728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3460170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3460740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3461209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3461417Z 
2023-01-11T22:55:24.3461531Z Running tests...
2023-01-11T22:55:24.3461929Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3462453Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3462927Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3463401Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85263
2023-01-11T22:55:24.3463852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85264
2023-01-11T22:55:24.3464451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3464880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3465445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3465913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3466466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3466905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3467470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3468011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3468426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3468893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3469380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3469901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3470576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3471261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3472182Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3472897Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3473724Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3474428Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3475262Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3475960Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3476781Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3477476Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3478312Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3479009Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3480204Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3480903Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3481740Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3482446Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3483285Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3483964Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3484380Z ok (6.703s)
2023-01-11T22:55:24.3484529Z 
2023-01-11T22:55:24.3484799Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3485127Z Ran 1 test in 6.703s
2023-01-11T22:55:24.3485269Z 
2023-01-11T22:55:24.3485362Z OK
2023-01-11T22:55:24.3485497Z 
2023-01-11T22:55:24.3485621Z Generating XML reports...
2023-01-11T22:55:24.3486179Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224059.xml
2023-01-11T22:55:24.3486900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3487353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3487932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3488398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3488608Z 
2023-01-11T22:55:24.3488722Z Running tests...
2023-01-11T22:55:24.3489122Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3489642Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3490113Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3490585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85378
2023-01-11T22:55:24.3491031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85379
2023-01-11T22:55:24.3491633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3492060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3492630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3493107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3493668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3494152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3494722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3495181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3495605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3496078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3496563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3497053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3497696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3498380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3499301Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3500011Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3500848Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3501620Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3502465Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3503160Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3504033Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3504750Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3505074Z ok (6.742s)
2023-01-11T22:55:24.3505220Z 
2023-01-11T22:55:24.3505484Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3505797Z Ran 1 test in 6.742s
2023-01-11T22:55:24.3505955Z 
2023-01-11T22:55:24.3506049Z OK
2023-01-11T22:55:24.3506181Z 
2023-01-11T22:55:24.3506304Z Generating XML reports...
2023-01-11T22:55:24.3506842Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224108.xml
2023-01-11T22:55:24.3507513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3507964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3508537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3508988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3509215Z 
2023-01-11T22:55:24.3509321Z Running tests...
2023-01-11T22:55:24.3509717Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3510234Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3510734Z   test_reduce_scatter_tensor_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3511215Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85493
2023-01-11T22:55:24.3511658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85494
2023-01-11T22:55:24.3512246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3512690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3513263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3513725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3514284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3514728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3515296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3515739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3516174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3516649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3517134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3517604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3518255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3519004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3520344Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3521186Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3522052Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3522756Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3523600Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3524282Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3525117Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3525814Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3526138Z ok (6.722s)
2023-01-11T22:55:24.3526286Z 
2023-01-11T22:55:24.3526533Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3526856Z Ran 1 test in 6.722s
2023-01-11T22:55:24.3527019Z 
2023-01-11T22:55:24.3527112Z OK
2023-01-11T22:55:24.3527242Z 
2023-01-11T22:55:24.3527368Z Generating XML reports...
2023-01-11T22:55:24.3527908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224117.xml
2023-01-11T22:55:24.3528287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3528462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3528848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3529039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3529059Z 
2023-01-11T22:55:24.3529165Z Running tests...
2023-01-11T22:55:24.3529426Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3529718Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3529972Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3530187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85608
2023-01-11T22:55:24.3530397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85609
2023-01-11T22:55:24.3530765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3530944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3531328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3531520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3531887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3532117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3532500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3532695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3532926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3533158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3533457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3533703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3534105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3534484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3535120Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3535404Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3536025Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3536312Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3536939Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3537220Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3537836Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3538114Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3538219Z ok (6.727s)
2023-01-11T22:55:24.3538238Z 
2023-01-11T22:55:24.3538504Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3538599Z Ran 1 test in 6.727s
2023-01-11T22:55:24.3538638Z 
2023-01-11T22:55:24.3538714Z OK
2023-01-11T22:55:24.3538733Z 
2023-01-11T22:55:24.3538859Z Generating XML reports...
2023-01-11T22:55:24.3539265Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224126.xml
2023-01-11T22:55:24.3539645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3539803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3540182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3540382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3540402Z 
2023-01-11T22:55:24.3540510Z Running tests...
2023-01-11T22:55:24.3540754Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3541066Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3541301Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3541577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85723
2023-01-11T22:55:24.3541787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85724
2023-01-11T22:55:24.3542158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3542332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3542749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3542928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3543299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3543471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3543849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3544037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3544269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3544497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3544744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3544985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3545371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3545765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3546399Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3546685Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3547305Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:55:24.3547583Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:55:24.3547686Z ok (6.619s)
2023-01-11T22:55:24.3547706Z 
2023-01-11T22:55:24.3547967Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3548081Z Ran 1 test in 6.619s
2023-01-11T22:55:24.3548104Z 
2023-01-11T22:55:24.3548197Z OK
2023-01-11T22:55:24.3548217Z 
2023-01-11T22:55:24.3548324Z Generating XML reports...
2023-01-11T22:55:24.3548725Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224135.xml
2023-01-11T22:55:24.3549092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3549264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3549648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3549835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3549855Z 
2023-01-11T22:55:24.3549965Z Running tests...
2023-01-11T22:55:24.3550228Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3550549Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3550869Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3551101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85840
2023-01-11T22:55:24.3551318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85841
2023-01-11T22:55:24.3551675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3551893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3552280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3552469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3552838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3552999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3553377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3553566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3553794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3554040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3554269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3554510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3554912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3555290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3555553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxw0nmmkf
2023-01-11T22:55:24.3555831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxw0nmmkf/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3556078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts5x23vg
2023-01-11T22:55:24.3556340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts5x23vg/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3556574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3556809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3556912Z ok (7.127s)
2023-01-11T22:55:24.3556932Z 
2023-01-11T22:55:24.3557195Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3557294Z Ran 1 test in 7.127s
2023-01-11T22:55:24.3557313Z 
2023-01-11T22:55:24.3557406Z OK
2023-01-11T22:55:24.3557425Z 
2023-01-11T22:55:24.3557549Z Generating XML reports...
2023-01-11T22:55:24.3558007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224144.xml
2023-01-11T22:55:24.3558381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3558557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3558940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3559131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3559151Z 
2023-01-11T22:55:24.3559256Z Running tests...
2023-01-11T22:55:24.3559855Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3560425Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3560731Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3560951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85961
2023-01-11T22:55:24.3561163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85962
2023-01-11T22:55:24.3561602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3561784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3562168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3562341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3562705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3562885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3563261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3563450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3563680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3563926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3564155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3564399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3564783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3565185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3565446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvas4v93v
2023-01-11T22:55:24.3565712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvas4v93v/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3565967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptyj7hpld
2023-01-11T22:55:24.3566235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptyj7hpld/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3566471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3566709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3566793Z ok (7.134s)
2023-01-11T22:55:24.3566833Z 
2023-01-11T22:55:24.3567087Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3567198Z Ran 1 test in 7.135s
2023-01-11T22:55:24.3567217Z 
2023-01-11T22:55:24.3567311Z OK
2023-01-11T22:55:24.3567330Z 
2023-01-11T22:55:24.3567454Z Generating XML reports...
2023-01-11T22:55:24.3567917Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224154.xml
2023-01-11T22:55:24.3568292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3568467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3568845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3569020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3569040Z 
2023-01-11T22:55:24.3569150Z Running tests...
2023-01-11T22:55:24.3569474Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3569785Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3570069Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3570289Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86082
2023-01-11T22:55:24.3570500Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86083
2023-01-11T22:55:24.3570916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3571079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3571464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3571654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3572025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3572199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3572573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3572762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3572997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3573225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3573450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3573691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3574093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3574487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3574746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2hfcijqy
2023-01-11T22:55:24.3575019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2hfcijqy/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3575271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zml5npi
2023-01-11T22:55:24.3575535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zml5npi/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3575772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3575988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3576095Z ok (7.137s)
2023-01-11T22:55:24.3576115Z 
2023-01-11T22:55:24.3576380Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3576492Z Ran 1 test in 7.137s
2023-01-11T22:55:24.3576512Z 
2023-01-11T22:55:24.3576605Z OK
2023-01-11T22:55:24.3576624Z 
2023-01-11T22:55:24.3576748Z Generating XML reports...
2023-01-11T22:55:24.3577208Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224203.xml
2023-01-11T22:55:24.3577586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3577744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3578130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3578322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3578401Z 
2023-01-11T22:55:24.3578515Z Running tests...
2023-01-11T22:55:24.3578780Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3579093Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3579394Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3579613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86203
2023-01-11T22:55:24.3579870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86204
2023-01-11T22:55:24.3580235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3580409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3580784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3580979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3581341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3581515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3581895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3582089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3582301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3582547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3582775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3583023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3583487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3583886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3584142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj1ax1bw
2023-01-11T22:55:24.3584417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj1ax1bw/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3584666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5awpljig
2023-01-11T22:55:24.3584910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5awpljig/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3585145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3585386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3585489Z ok (7.126s)
2023-01-11T22:55:24.3585509Z 
2023-01-11T22:55:24.3585774Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3585887Z Ran 1 test in 7.126s
2023-01-11T22:55:24.3585906Z 
2023-01-11T22:55:24.3585999Z OK
2023-01-11T22:55:24.3586018Z 
2023-01-11T22:55:24.3586142Z Generating XML reports...
2023-01-11T22:55:24.3586587Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224213.xml
2023-01-11T22:55:24.3586960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3587135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3587510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3587760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3587780Z 
2023-01-11T22:55:24.3587887Z Running tests...
2023-01-11T22:55:24.3588153Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3588469Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3588752Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3588995Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86324
2023-01-11T22:55:24.3589216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86325
2023-01-11T22:55:24.3589590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3589766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3590144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3590336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3590701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3590876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3591257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3591430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3591659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3591906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3592133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3592378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3592781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3593338Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.3593734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3594308Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.3594572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8kkqaqp
2023-01-11T22:55:24.3594843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8kkqaqp/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3595083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptj0m75_v
2023-01-11T22:55:24.3595345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptj0m75_v/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3595448Z ok (6.535s)
2023-01-11T22:55:24.3595468Z 
2023-01-11T22:55:24.3595736Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3595849Z Ran 1 test in 6.535s
2023-01-11T22:55:24.3595869Z 
2023-01-11T22:55:24.3596014Z OK
2023-01-11T22:55:24.3596034Z 
2023-01-11T22:55:24.3596160Z Generating XML reports...
2023-01-11T22:55:24.3596627Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224222.xml
2023-01-11T22:55:24.3596977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3597153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3597575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3597772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3597792Z 
2023-01-11T22:55:24.3597901Z Running tests...
2023-01-11T22:55:24.3598165Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3598478Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3598761Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3598982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86445
2023-01-11T22:55:24.3599176Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86446
2023-01-11T22:55:24.3599798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3599973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3600366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3600550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3600917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3601093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3601471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3601642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3601871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3602113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3602344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3602587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3602988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3603535Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.3603934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3604486Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.3604742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfubzs1ja
2023-01-11T22:55:24.3605015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfubzs1ja/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3605333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu8w94ty
2023-01-11T22:55:24.3605596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu8w94ty/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3605699Z ok (6.706s)
2023-01-11T22:55:24.3605720Z 
2023-01-11T22:55:24.3605992Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3606104Z Ran 1 test in 6.707s
2023-01-11T22:55:24.3606124Z 
2023-01-11T22:55:24.3606214Z OK
2023-01-11T22:55:24.3606292Z 
2023-01-11T22:55:24.3606422Z Generating XML reports...
2023-01-11T22:55:24.3606891Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224231.xml
2023-01-11T22:55:24.3607265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3607428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3607806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3607993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3608014Z 
2023-01-11T22:55:24.3608122Z Running tests...
2023-01-11T22:55:24.3608382Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3608694Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3608977Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3609198Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86566
2023-01-11T22:55:24.3609392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86567
2023-01-11T22:55:24.3609760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3609935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3610313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3610500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3610868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3611044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3611423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3611614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3611829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3612076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3612301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3612549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3612951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3613349Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3613610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxlsz9e5
2023-01-11T22:55:24.3613880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxlsz9e5/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3614113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s47hold
2023-01-11T22:55:24.3614438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s47hold/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3614540Z ok (6.619s)
2023-01-11T22:55:24.3614560Z 
2023-01-11T22:55:24.3614827Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3614938Z Ran 1 test in 6.619s
2023-01-11T22:55:24.3614958Z 
2023-01-11T22:55:24.3615051Z OK
2023-01-11T22:55:24.3615070Z 
2023-01-11T22:55:24.3615197Z Generating XML reports...
2023-01-11T22:55:24.3615705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224240.xml
2023-01-11T22:55:24.3616086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3616243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3616621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3616817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3616837Z 
2023-01-11T22:55:24.3616946Z Running tests...
2023-01-11T22:55:24.3617207Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3617516Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3617812Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3618029Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86687
2023-01-11T22:55:24.3618222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86688
2023-01-11T22:55:24.3618596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3618771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3619156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3619346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3619708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3619880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3620259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3620448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3620662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3620908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3621139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3621381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3621784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3622177Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3622438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42hgx9o6
2023-01-11T22:55:24.3622705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42hgx9o6/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3622960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7djgncta
2023-01-11T22:55:24.3623211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7djgncta/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3623392Z ok (6.615s)
2023-01-11T22:55:24.3623412Z 
2023-01-11T22:55:24.3623677Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3623789Z Ran 1 test in 6.615s
2023-01-11T22:55:24.3623809Z 
2023-01-11T22:55:24.3623904Z OK
2023-01-11T22:55:24.3623924Z 
2023-01-11T22:55:24.3624048Z Generating XML reports...
2023-01-11T22:55:24.3624510Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224249.xml
2023-01-11T22:55:24.3624923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3625086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3625468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3625657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3625682Z 
2023-01-11T22:55:24.3625791Z Running tests...
2023-01-11T22:55:24.3626053Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3626361Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3626633Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3626853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86808
2023-01-11T22:55:24.3627071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86809
2023-01-11T22:55:24.3627421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3627595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3627970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3628161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3628528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3628700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3629075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3629270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3629482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3629727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3629957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3630204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3630607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3631003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3631105Z ok (6.615s)
2023-01-11T22:55:24.3631125Z 
2023-01-11T22:55:24.3631384Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3631501Z Ran 1 test in 6.615s
2023-01-11T22:55:24.3631522Z 
2023-01-11T22:55:24.3631596Z OK
2023-01-11T22:55:24.3631614Z 
2023-01-11T22:55:24.3631739Z Generating XML reports...
2023-01-11T22:55:24.3632201Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224258.xml
2023-01-11T22:55:24.3632570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3632806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3633184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3633373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3633393Z 
2023-01-11T22:55:24.3633504Z Running tests...
2023-01-11T22:55:24.3633748Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3634103Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3634339Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3634711Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3634927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86925
2023-01-11T22:55:24.3635144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86926
2023-01-11T22:55:24.3635513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3635687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3636064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3636241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3636607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3636779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3637156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3637351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3637582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3637829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3638061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3638281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3638687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3639083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3639339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qgte_fz
2023-01-11T22:55:24.3640053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qgte_fz/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3640367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzgqmm3jb
2023-01-11T22:55:24.3640633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzgqmm3jb/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3640737Z ok (6.015s)
2023-01-11T22:55:24.3640758Z 
2023-01-11T22:55:24.3641040Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3641141Z Ran 1 test in 6.016s
2023-01-11T22:55:24.3641181Z 
2023-01-11T22:55:24.3641257Z OK
2023-01-11T22:55:24.3641276Z 
2023-01-11T22:55:24.3641401Z Generating XML reports...
2023-01-11T22:55:24.3641866Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224307.xml
2023-01-11T22:55:24.3642235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3642509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3642893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3643086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3643106Z 
2023-01-11T22:55:24.3643216Z Running tests...
2023-01-11T22:55:24.3643461Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3643835Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3644090Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3644359Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3644579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87045
2023-01-11T22:55:24.3644797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87046
2023-01-11T22:55:24.3645175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3645352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3645714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3645906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3646275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3646451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3646824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3647019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3647246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3647489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3647717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3647942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3648347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3648745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3649001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xi6zs9l
2023-01-11T22:55:24.3649261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xyyrffr
2023-01-11T22:55:24.3649530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xi6zs9l/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3649796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xyyrffr/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3649899Z ok (6.030s)
2023-01-11T22:55:24.3649918Z 
2023-01-11T22:55:24.3650186Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3650281Z Ran 1 test in 6.031s
2023-01-11T22:55:24.3650304Z 
2023-01-11T22:55:24.3650398Z OK
2023-01-11T22:55:24.3650418Z 
2023-01-11T22:55:24.3650542Z Generating XML reports...
2023-01-11T22:55:24.3651006Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224315.xml
2023-01-11T22:55:24.3651378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3651608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3651992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3652186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3652206Z 
2023-01-11T22:55:24.3652297Z Running tests...
2023-01-11T22:55:24.3652556Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3652917Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3653164Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3653417Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3653637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87165
2023-01-11T22:55:24.3653853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87166
2023-01-11T22:55:24.3654228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3654385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3654761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3654952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3655337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3655803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3656368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3656817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3657269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3657757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3658245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3658716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3659377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3660065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3660596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16m16msh
2023-01-11T22:55:24.3661139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16m16msh/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3661673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxhp62kno
2023-01-11T22:55:24.3662210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxhp62kno/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3662710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3663191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3663671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3664143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3665284Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3666079Z   warnings.warn(
2023-01-11T22:55:24.3667187Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3667927Z   warnings.warn(
2023-01-11T22:55:24.3668304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3668767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3669245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3669719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3670181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3670671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3671165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3671647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3671990Z ok (6.117s)
2023-01-11T22:55:24.3672120Z 
2023-01-11T22:55:24.3672394Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3672721Z Ran 1 test in 6.118s
2023-01-11T22:55:24.3672881Z 
2023-01-11T22:55:24.3672975Z OK
2023-01-11T22:55:24.3673109Z 
2023-01-11T22:55:24.3675197Z Generating XML reports...
2023-01-11T22:55:24.3675828Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224323.xml
2023-01-11T22:55:24.3676555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3677005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3677557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3678020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3678253Z 
2023-01-11T22:55:24.3678364Z Running tests...
2023-01-11T22:55:24.3678750Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3679272Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3679940Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3704344Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3704919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87285
2023-01-11T22:55:24.3705407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87286
2023-01-11T22:55:24.3706102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3706557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3707170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3707659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3708252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3708743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3709539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3710033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3710470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3710984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3711569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3712091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3712769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3713491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3714069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_z2mf4_
2023-01-11T22:55:24.3714625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_z2mf4_/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3715160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdkef_dx
2023-01-11T22:55:24.3715716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdkef_dx/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3716253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3716909Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3717430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3717938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3719182Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3720485Z   warnings.warn(
2023-01-11T22:55:24.3721542Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3722265Z   warnings.warn(
2023-01-11T22:55:24.3722637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3723119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3723577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3724051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3724527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3724979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3725448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3725950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3726285Z ok (6.123s)
2023-01-11T22:55:24.3726416Z 
2023-01-11T22:55:24.3726695Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3727159Z Ran 1 test in 6.124s
2023-01-11T22:55:24.3727296Z 
2023-01-11T22:55:24.3727389Z OK
2023-01-11T22:55:24.3727521Z 
2023-01-11T22:55:24.3727628Z Generating XML reports...
2023-01-11T22:55:24.3728247Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224332.xml
2023-01-11T22:55:24.3728998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3729454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3730067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3730547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3730777Z 
2023-01-11T22:55:24.3730902Z Running tests...
2023-01-11T22:55:24.3731313Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3731831Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3732341Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3732963Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3733443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87405
2023-01-11T22:55:24.3733871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87406
2023-01-11T22:55:24.3734470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3734931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3735491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3735938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3736517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3736955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3737500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3737958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3738396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3738883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3739342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3739818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3740481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3741164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3741687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3b6nnnw5
2023-01-11T22:55:24.3742223Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3b6nnnw5/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3742755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5uiywqn
2023-01-11T22:55:24.3743270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5uiywqn/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3743774Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3744246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3744784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3745243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3745585Z ok (6.131s)
2023-01-11T22:55:24.3745735Z 
2023-01-11T22:55:24.3746006Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3746314Z Ran 1 test in 6.131s
2023-01-11T22:55:24.3746473Z 
2023-01-11T22:55:24.3746568Z OK
2023-01-11T22:55:24.3746701Z 
2023-01-11T22:55:24.3746876Z Generating XML reports...
2023-01-11T22:55:24.3747500Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224340.xml
2023-01-11T22:55:24.3748202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3748648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3749222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3749667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3749893Z 
2023-01-11T22:55:24.3750003Z Running tests...
2023-01-11T22:55:24.3750399Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3750926Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3751416Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3752037Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3752526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87525
2023-01-11T22:55:24.3752954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87526
2023-01-11T22:55:24.3753566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3754016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3754582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3755031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3755608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3756049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3756615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3757058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3757491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3757981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3758444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3758922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3760069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3760783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3761304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mcd066r
2023-01-11T22:55:24.3761840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mcd066r/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3762465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ffetxlu
2023-01-11T22:55:24.3762996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ffetxlu/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3763491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3763976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3764447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3764962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3765341Z ok (6.029s)
2023-01-11T22:55:24.3765471Z 
2023-01-11T22:55:24.3765746Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3766073Z Ran 1 test in 6.029s
2023-01-11T22:55:24.3766215Z 
2023-01-11T22:55:24.3766307Z OK
2023-01-11T22:55:24.3766445Z 
2023-01-11T22:55:24.3766571Z Generating XML reports...
2023-01-11T22:55:24.3767206Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224349.xml
2023-01-11T22:55:24.3767886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3768332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3768907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3769370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3769577Z 
2023-01-11T22:55:24.3769684Z Running tests...
2023-01-11T22:55:24.3770082Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3770604Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3771075Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3771697Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3772193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87645
2023-01-11T22:55:24.3772637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87646
2023-01-11T22:55:24.3773217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3773662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3774230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3774671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3775232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3775667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3776228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3776687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3777093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3777581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3778068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3778516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3779173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3779974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3780480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4ab4282
2023-01-11T22:55:24.3781010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4ab4282/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3781534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8akbhrea
2023-01-11T22:55:24.3782107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8akbhrea/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3782610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3783084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3784101Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:55:24.3785605Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:55:24.3786575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3787051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3787380Z ok (6.118s)
2023-01-11T22:55:24.3787526Z 
2023-01-11T22:55:24.3787800Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3788123Z Ran 1 test in 6.118s
2023-01-11T22:55:24.3788283Z 
2023-01-11T22:55:24.3788358Z OK
2023-01-11T22:55:24.3788490Z 
2023-01-11T22:55:24.3788614Z Generating XML reports...
2023-01-11T22:55:24.3789231Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224357.xml
2023-01-11T22:55:24.3789948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3790374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3790948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3791400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3791627Z 
2023-01-11T22:55:24.3791741Z Running tests...
2023-01-11T22:55:24.3792123Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3792644Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3793128Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3793736Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3794281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87765
2023-01-11T22:55:24.3794726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87766
2023-01-11T22:55:24.3795393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3795818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3796389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3796848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3797401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3797880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3798440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3798878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3799284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3799993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3800466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3800911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3801540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3802213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3802755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcp4c9lmx
2023-01-11T22:55:24.3803263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcp4c9lmx/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3803779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtb46vzd
2023-01-11T22:55:24.3804312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtb46vzd/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3804665Z ok (6.008s)
2023-01-11T22:55:24.3804795Z 
2023-01-11T22:55:24.3805055Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3805359Z Ran 1 test in 6.009s
2023-01-11T22:55:24.3805506Z 
2023-01-11T22:55:24.3805588Z OK
2023-01-11T22:55:24.3805709Z 
2023-01-11T22:55:24.3805815Z Generating XML reports...
2023-01-11T22:55:24.3806406Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224406.xml
2023-01-11T22:55:24.3807097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3807520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3808058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3808503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3808718Z 
2023-01-11T22:55:24.3808816Z Running tests...
2023-01-11T22:55:24.3809192Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3809709Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3810177Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3810695Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3811168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87885
2023-01-11T22:55:24.3811614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87886
2023-01-11T22:55:24.3812219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3812733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3813309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3813771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3814339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3814830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3815407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3815863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3816300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3816772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3817250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3817725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3818350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3819035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3819579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxczbkjkz
2023-01-11T22:55:24.3820109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxczbkjkz/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3820617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_2alf85
2023-01-11T22:55:24.3821153Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_2alf85/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3821655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3822129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3822578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3823053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3823387Z ok (6.019s)
2023-01-11T22:55:24.3823529Z 
2023-01-11T22:55:24.3823785Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3824099Z Ran 1 test in 6.020s
2023-01-11T22:55:24.3824259Z 
2023-01-11T22:55:24.3824343Z OK
2023-01-11T22:55:24.3824474Z 
2023-01-11T22:55:24.3824580Z Generating XML reports...
2023-01-11T22:55:24.3825189Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224414.xml
2023-01-11T22:55:24.3825892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3826323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3826872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3827328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3827546Z 
2023-01-11T22:55:24.3827653Z Running tests...
2023-01-11T22:55:24.3828045Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3828542Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3829029Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3829617Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3830086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88005
2023-01-11T22:55:24.3830517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88006
2023-01-11T22:55:24.3831113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3831602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3832156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3832615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3833171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3833586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3834143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3834588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3835019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3835457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3835932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3836416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3837054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3837724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3838255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplk2eqc72
2023-01-11T22:55:24.3838780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplk2eqc72/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3839287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0kgjdeg
2023-01-11T22:55:24.3840332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0kgjdeg/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3841378Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:55:24.3842879Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:55:24.3844535Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3845335Z   warnings.warn(
2023-01-11T22:55:24.3846369Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3847133Z   warnings.warn(
2023-01-11T22:55:24.3847495Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3847969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3848434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3848888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3849225Z ok (6.133s)
2023-01-11T22:55:24.3849370Z 
2023-01-11T22:55:24.3849640Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3849942Z Ran 1 test in 6.133s
2023-01-11T22:55:24.3850103Z 
2023-01-11T22:55:24.3850197Z OK
2023-01-11T22:55:24.3850323Z 
2023-01-11T22:55:24.3850445Z Generating XML reports...
2023-01-11T22:55:24.3851053Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224422.xml
2023-01-11T22:55:24.3851755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3852190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3852751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3853213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3853425Z 
2023-01-11T22:55:24.3853527Z Running tests...
2023-01-11T22:55:24.3853914Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3854436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3854906Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3855440Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3855923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88125
2023-01-11T22:55:24.3856361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88126
2023-01-11T22:55:24.3856946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3857385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3857953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3858396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3858955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3859390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3859952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3860393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3860820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3861301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3861833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3862283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3862931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3863610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3864177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfutj1ss7
2023-01-11T22:55:24.3864717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfutj1ss7/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3865237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qsmwz5d
2023-01-11T22:55:24.3865761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qsmwz5d/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3866932Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3867642Z   warnings.warn(
2023-01-11T22:55:24.3868677Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:55:24.3869381Z   warnings.warn(
2023-01-11T22:55:24.3869736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3870228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3870667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3871129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3871454Z ok (6.134s)
2023-01-11T22:55:24.3871595Z 
2023-01-11T22:55:24.3871862Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3872187Z Ran 1 test in 6.135s
2023-01-11T22:55:24.3872347Z 
2023-01-11T22:55:24.3872424Z OK
2023-01-11T22:55:24.3872550Z 
2023-01-11T22:55:24.3872674Z Generating XML reports...
2023-01-11T22:55:24.3873277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224431.xml
2023-01-11T22:55:24.3873990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3874422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3874986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3875439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3875667Z 
2023-01-11T22:55:24.3875772Z Running tests...
2023-01-11T22:55:24.3876155Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3876679Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3877174Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3877653Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3878107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88245
2023-01-11T22:55:24.3878609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88246
2023-01-11T22:55:24.3879209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3879866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3880428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3880952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3881521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3881947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3882507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3882983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3883423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3883909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3884389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3884826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3885483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3886146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3886680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6x9nnn80
2023-01-11T22:55:24.3887192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6x9nnn80/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3887707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppasda7nf
2023-01-11T22:55:24.3888235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppasda7nf/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3888747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3889212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3889687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3890155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3890487Z ok (6.032s)
2023-01-11T22:55:24.3890631Z 
2023-01-11T22:55:24.3890901Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3891226Z Ran 1 test in 6.032s
2023-01-11T22:55:24.3891388Z 
2023-01-11T22:55:24.3891484Z OK
2023-01-11T22:55:24.3891599Z 
2023-01-11T22:55:24.3891716Z Generating XML reports...
2023-01-11T22:55:24.3892326Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224439.xml
2023-01-11T22:55:24.3893036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3893479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3894038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3894542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3894772Z 
2023-01-11T22:55:24.3894877Z Running tests...
2023-01-11T22:55:24.3895264Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3895889Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3896385Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3896888Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3897338Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88365
2023-01-11T22:55:24.3897787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88366
2023-01-11T22:55:24.3898439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3898875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3899453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3899919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3900498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3900918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3901487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3901949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3902400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3902866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3903348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3903825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3904464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3905149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3905687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxog_mgv5
2023-01-11T22:55:24.3906220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxog_mgv5/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3906732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso45p0xs
2023-01-11T22:55:24.3907265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso45p0xs/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3907777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3908258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3908721Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3909201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3909670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3910117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3910589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3911062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3911402Z ok (6.129s)
2023-01-11T22:55:24.3911531Z 
2023-01-11T22:55:24.3911810Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3912134Z Ran 1 test in 6.129s
2023-01-11T22:55:24.3912293Z 
2023-01-11T22:55:24.3912390Z OK
2023-01-11T22:55:24.3912581Z 
2023-01-11T22:55:24.3912688Z Generating XML reports...
2023-01-11T22:55:24.3913306Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224448.xml
2023-01-11T22:55:24.3914020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3914464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3915060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3915530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3915755Z 
2023-01-11T22:55:24.3915862Z Running tests...
2023-01-11T22:55:24.3916249Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3916773Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3917308Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3917817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88485
2023-01-11T22:55:24.3918242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88486
2023-01-11T22:55:24.3918837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3919278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3920371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3920815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3921387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3921836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3922388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3922849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3923280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3923767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3924233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3924710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3925363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3926046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3926566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v_hhvzo
2023-01-11T22:55:24.3927098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v_hhvzo/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3927626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpea9pn7lg
2023-01-11T22:55:24.3928157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpea9pn7lg/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3928524Z ok (6.636s)
2023-01-11T22:55:24.3928671Z 
2023-01-11T22:55:24.3928939Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3929262Z Ran 1 test in 6.636s
2023-01-11T22:55:24.3929420Z 
2023-01-11T22:55:24.3929497Z OK
2023-01-11T22:55:24.3929636Z 
2023-01-11T22:55:24.3929758Z Generating XML reports...
2023-01-11T22:55:24.3930367Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224456.xml
2023-01-11T22:55:24.3931197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3931627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3932190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3932652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3932954Z 
2023-01-11T22:55:24.3933057Z Running tests...
2023-01-11T22:55:24.3933462Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3933982Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3934522Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3935028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88606
2023-01-11T22:55:24.3935475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88607
2023-01-11T22:55:24.3936076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3936504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3937077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3937543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3938111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3938536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3939103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3939565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3939796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3940037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3940246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3940492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3940896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3941294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3941551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6t07wxa
2023-01-11T22:55:24.3941816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6t07wxa/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3942070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3chvi4f
2023-01-11T22:55:24.3942333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3chvi4f/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3942435Z ok (6.614s)
2023-01-11T22:55:24.3942456Z 
2023-01-11T22:55:24.3942706Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3942823Z Ran 1 test in 6.614s
2023-01-11T22:55:24.3942843Z 
2023-01-11T22:55:24.3942934Z OK
2023-01-11T22:55:24.3942953Z 
2023-01-11T22:55:24.3943077Z Generating XML reports...
2023-01-11T22:55:24.3943538Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224505.xml
2023-01-11T22:55:24.3943977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3944151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3944528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3944723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3944744Z 
2023-01-11T22:55:24.3944834Z Running tests...
2023-01-11T22:55:24.3945142Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3945459Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3945768Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3945983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88727
2023-01-11T22:55:24.3946201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88728
2023-01-11T22:55:24.3946572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3946744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3947102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3947294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3947664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3947840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3948211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3948404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3948633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3948875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3949099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3949325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3949730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3950131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3950383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5nouceo
2023-01-11T22:55:24.3950650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5nouceo/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3950906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgi7h396b
2023-01-11T22:55:24.3951172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgi7h396b/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3951272Z ok (6.633s)
2023-01-11T22:55:24.3951292Z 
2023-01-11T22:55:24.3951560Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3951654Z Ran 1 test in 6.634s
2023-01-11T22:55:24.3951673Z 
2023-01-11T22:55:24.3951768Z OK
2023-01-11T22:55:24.3951788Z 
2023-01-11T22:55:24.3951912Z Generating XML reports...
2023-01-11T22:55:24.3952370Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224514.xml
2023-01-11T22:55:24.3952736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3952970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3953349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3953544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3953563Z 
2023-01-11T22:55:24.3953671Z Running tests...
2023-01-11T22:55:24.3953914Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3954272Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3954515Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3954801Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3955016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88848
2023-01-11T22:55:24.3955233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88849
2023-01-11T22:55:24.3955606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3955779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3956126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3956301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3956681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3956867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3957241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3957426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3957654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3957901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3958125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3958343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3958749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3959141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3959763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppiwrgrc3
2023-01-11T22:55:24.3960202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppiwrgrc3/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3960460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd0m7odh9
2023-01-11T22:55:24.3960730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd0m7odh9/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3960830Z ok (6.603s)
2023-01-11T22:55:24.3960850Z 
2023-01-11T22:55:24.3961125Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3961221Z Ran 1 test in 6.604s
2023-01-11T22:55:24.3961240Z 
2023-01-11T22:55:24.3961336Z OK
2023-01-11T22:55:24.3961355Z 
2023-01-11T22:55:24.3961477Z Generating XML reports...
2023-01-11T22:55:24.3961942Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224523.xml
2023-01-11T22:55:24.3962314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3962619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3963005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3963193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3963212Z 
2023-01-11T22:55:24.3963303Z Running tests...
2023-01-11T22:55:24.3963565Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3963942Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3964180Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3964469Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3964690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88969
2023-01-11T22:55:24.3964908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88970
2023-01-11T22:55:24.3965283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3965454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3965811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3965999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3966366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3966534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3966907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3967091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3967321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3967569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3967776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3968012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3968416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3968812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3969064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvh44ij_5
2023-01-11T22:55:24.3969327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvh44ij_5/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3969584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebiy5x8q
2023-01-11T22:55:24.3969847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebiy5x8q/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3969953Z ok (6.697s)
2023-01-11T22:55:24.3969973Z 
2023-01-11T22:55:24.3970238Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3970334Z Ran 1 test in 6.698s
2023-01-11T22:55:24.3970353Z 
2023-01-11T22:55:24.3970447Z OK
2023-01-11T22:55:24.3970466Z 
2023-01-11T22:55:24.3970588Z Generating XML reports...
2023-01-11T22:55:24.3971047Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224532.xml
2023-01-11T22:55:24.3971421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3971656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3972034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3972227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3972246Z 
2023-01-11T22:55:24.3972337Z Running tests...
2023-01-11T22:55:24.3972597Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3972950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3973242Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3973462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89090
2023-01-11T22:55:24.3973677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89091
2023-01-11T22:55:24.3974051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3974226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3974604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3974777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3975136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3975309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3975683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3975869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3976099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3976328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3976475Z skip: Need at least 4 CUDA devices (3.818s)
2023-01-11T22:55:24.3976495Z 
2023-01-11T22:55:24.3976763Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3976858Z Ran 1 test in 3.818s
2023-01-11T22:55:24.3976877Z 
2023-01-11T22:55:24.3976985Z OK (skipped=1)
2023-01-11T22:55:24.3977004Z 
2023-01-11T22:55:24.3977126Z Generating XML reports...
2023-01-11T22:55:24.3977586Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224541.xml
2023-01-11T22:55:24.3977951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3978127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3978498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3978694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3978714Z 
2023-01-11T22:55:24.3978805Z Running tests...
2023-01-11T22:55:24.3979068Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3979380Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3979582Z   test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.3979839Z Tests that DDP with ``device_ids`` specified can run a forward and ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3980058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89193
2023-01-11T22:55:24.3980270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89194
2023-01-11T22:55:24.3980641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3980871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3981236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3981428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3981786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3982000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3982384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3982567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3982795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3983045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3983250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3983489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3983890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3984286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3984539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74bpd5ga
2023-01-11T22:55:24.3984807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74bpd5ga/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3985056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm930l93w
2023-01-11T22:55:24.3985323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm930l93w/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3985425Z ok (7.445s)
2023-01-11T22:55:24.3985445Z 
2023-01-11T22:55:24.3985691Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3985808Z Ran 1 test in 7.445s
2023-01-11T22:55:24.3985827Z 
2023-01-11T22:55:24.3985920Z OK
2023-01-11T22:55:24.3985939Z 
2023-01-11T22:55:24.3986060Z Generating XML reports...
2023-01-11T22:55:24.3986521Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224547.xml
2023-01-11T22:55:24.3986884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3987066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3987439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3987632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3987652Z 
2023-01-11T22:55:24.3987743Z Running tests...
2023-01-11T22:55:24.3988004Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3988310Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3988576Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3988800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89314
2023-01-11T22:55:24.3989014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89315
2023-01-11T22:55:24.3989383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3989555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3989995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3990168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3990523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3990699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3991120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3991312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3991543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.3991788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.3992007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.3992234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.3992635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3993030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.3993287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0sxks64
2023-01-11T22:55:24.3993558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0sxks64/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3993815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7ubpw_i
2023-01-11T22:55:24.3994087Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7ubpw_i/_remote_module_non_scriptable.py
2023-01-11T22:55:24.3994375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3994611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3994823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3995052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3995289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3995520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3995749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3995972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.3996072Z ok (6.926s)
2023-01-11T22:55:24.3996091Z 
2023-01-11T22:55:24.3996362Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3996473Z Ran 1 test in 6.926s
2023-01-11T22:55:24.3996494Z 
2023-01-11T22:55:24.3996570Z OK
2023-01-11T22:55:24.3996589Z 
2023-01-11T22:55:24.3996711Z Generating XML reports...
2023-01-11T22:55:24.3997174Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224557.xml
2023-01-11T22:55:24.3997546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.3997725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.3998106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.3998293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.3998313Z 
2023-01-11T22:55:24.3998419Z Running tests...
2023-01-11T22:55:24.3998725Z ----------------------------------------------------------------------
2023-01-11T22:55:24.3999038Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.3999315Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.3999755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89435
2023-01-11T22:55:24.3999988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89436
2023-01-11T22:55:24.4000444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4000629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4001013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4001199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4001549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4001720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4002095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4002284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4002517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4002760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4002985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4003220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4003628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4004147Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2023-01-11T22:55:24.4004422Z   warnings.warn('Lazy modules are a new feature under heavy development '
2023-01-11T22:55:24.4004817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4005352Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2023-01-11T22:55:24.4005618Z   warnings.warn('Lazy modules are a new feature under heavy development '
2023-01-11T22:55:24.4005875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr01geaxh
2023-01-11T22:55:24.4006145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr01geaxh/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4006394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wdxaohk
2023-01-11T22:55:24.4006664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wdxaohk/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4006764Z ok (3.922s)
2023-01-11T22:55:24.4006785Z 
2023-01-11T22:55:24.4007032Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4007151Z Ran 1 test in 3.922s
2023-01-11T22:55:24.4007171Z 
2023-01-11T22:55:24.4007262Z OK
2023-01-11T22:55:24.4007282Z 
2023-01-11T22:55:24.4007403Z Generating XML reports...
2023-01-11T22:55:24.4007862Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224606.xml
2023-01-11T22:55:24.4008227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4008481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4008863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4009035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4009076Z 
2023-01-11T22:55:24.4009167Z Running tests...
2023-01-11T22:55:24.4009426Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4009787Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4010069Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4010285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89542
2023-01-11T22:55:24.4010502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89543
2023-01-11T22:55:24.4010880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4011052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4011415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4011605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4011965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4012139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4012507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4012698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4012928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4013170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4013392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4013613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4014021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4014415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4014670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbri8fwuv
2023-01-11T22:55:24.4014936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbri8fwuv/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4015195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw6g438be
2023-01-11T22:55:24.4015455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw6g438be/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4015556Z ok (6.620s)
2023-01-11T22:55:24.4015576Z 
2023-01-11T22:55:24.4015844Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4015941Z Ran 1 test in 6.620s
2023-01-11T22:55:24.4015960Z 
2023-01-11T22:55:24.4016051Z OK
2023-01-11T22:55:24.4016070Z 
2023-01-11T22:55:24.4016193Z Generating XML reports...
2023-01-11T22:55:24.4016653Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224612.xml
2023-01-11T22:55:24.4017023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4017195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4017634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4017821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4017841Z 
2023-01-11T22:55:24.4017932Z Running tests...
2023-01-11T22:55:24.4018192Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4018501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4018834Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4019058Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89663
2023-01-11T22:55:24.4019275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89664
2023-01-11T22:55:24.4019644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4019822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4020203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4020376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4020736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4020915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4021285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4021476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4021700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4021945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4022173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4022391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4022790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4023190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4023445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbvft1g3n
2023-01-11T22:55:24.4023712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbvft1g3n/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4023966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngyymp21
2023-01-11T22:55:24.4024237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngyymp21/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4024339Z ok (6.606s)
2023-01-11T22:55:24.4024359Z 
2023-01-11T22:55:24.4024624Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4024718Z Ran 1 test in 6.606s
2023-01-11T22:55:24.4024755Z 
2023-01-11T22:55:24.4024831Z OK
2023-01-11T22:55:24.4024850Z 
2023-01-11T22:55:24.4024972Z Generating XML reports...
2023-01-11T22:55:24.4025432Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224621.xml
2023-01-11T22:55:24.4025801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4025976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4026355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4026605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4026625Z 
2023-01-11T22:55:24.4026733Z Running tests...
2023-01-11T22:55:24.4026978Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4027286Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4027550Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4027837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89784
2023-01-11T22:55:24.4028062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89785
2023-01-11T22:55:24.4028437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4028609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4028985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4029177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4029522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4029693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4030069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4030256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4030485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4030726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4030947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4031190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4031571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4031964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4032226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9gvm0qw
2023-01-11T22:55:24.4032494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9gvm0qw/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4032750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta7noxdd
2023-01-11T22:55:24.4033013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta7noxdd/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4033250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4033486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4033725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4033946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4034352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4034748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4034979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4035207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4035366Z ok (7.346s)
2023-01-11T22:55:24.4035387Z 
2023-01-11T22:55:24.4035659Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4035770Z Ran 1 test in 7.346s
2023-01-11T22:55:24.4035790Z 
2023-01-11T22:55:24.4035882Z OK
2023-01-11T22:55:24.4035901Z 
2023-01-11T22:55:24.4036006Z Generating XML reports...
2023-01-11T22:55:24.4036463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224630.xml
2023-01-11T22:55:24.4036882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4037068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4037452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4037642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4037665Z 
2023-01-11T22:55:24.4037773Z Running tests...
2023-01-11T22:55:24.4038040Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4038347Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4038630Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4039651Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2023-01-11T22:55:24.4039685Z 
2023-01-11T22:55:24.4040175Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4040297Z Ran 1 test in 1.609s
2023-01-11T22:55:24.4040318Z 
2023-01-11T22:55:24.4040432Z OK (skipped=1)
2023-01-11T22:55:24.4040460Z 
2023-01-11T22:55:24.4040575Z Generating XML reports...
2023-01-11T22:55:24.4041038Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224640.xml
2023-01-11T22:55:24.4041406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4041580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4041961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4042135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4042154Z 
2023-01-11T22:55:24.4042262Z Running tests...
2023-01-11T22:55:24.4042524Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4042832Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4043130Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4043876Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.618s)
2023-01-11T22:55:24.4043896Z 
2023-01-11T22:55:24.4044157Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4044269Z Ran 1 test in 1.618s
2023-01-11T22:55:24.4044288Z 
2023-01-11T22:55:24.4044395Z OK (skipped=1)
2023-01-11T22:55:24.4044413Z 
2023-01-11T22:55:24.4044519Z Generating XML reports...
2023-01-11T22:55:24.4044973Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224644.xml
2023-01-11T22:55:24.4045445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4045618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4045992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4046181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4046201Z 
2023-01-11T22:55:24.4046307Z Running tests...
2023-01-11T22:55:24.4046623Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4046947Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4047222Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4047963Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.602s)
2023-01-11T22:55:24.4047987Z 
2023-01-11T22:55:24.4048249Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4048360Z Ran 1 test in 1.602s
2023-01-11T22:55:24.4048379Z 
2023-01-11T22:55:24.4048486Z OK (skipped=1)
2023-01-11T22:55:24.4048505Z 
2023-01-11T22:55:24.4048627Z Generating XML reports...
2023-01-11T22:55:24.4049082Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224647.xml
2023-01-11T22:55:24.4049454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4049627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4050003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4050180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4050200Z 
2023-01-11T22:55:24.4050308Z Running tests...
2023-01-11T22:55:24.4050571Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4050879Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4051195Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4051938Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2023-01-11T22:55:24.4051961Z 
2023-01-11T22:55:24.4052223Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4052334Z Ran 1 test in 1.614s
2023-01-11T22:55:24.4052353Z 
2023-01-11T22:55:24.4052458Z OK (skipped=1)
2023-01-11T22:55:24.4052476Z 
2023-01-11T22:55:24.4052599Z Generating XML reports...
2023-01-11T22:55:24.4053036Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224651.xml
2023-01-11T22:55:24.4053407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4053584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4053961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4054153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4054172Z 
2023-01-11T22:55:24.4054279Z Running tests...
2023-01-11T22:55:24.4054605Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4054915Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4055208Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4055996Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2023-01-11T22:55:24.4056037Z 
2023-01-11T22:55:24.4056287Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4056399Z Ran 1 test in 1.625s
2023-01-11T22:55:24.4056418Z 
2023-01-11T22:55:24.4056525Z OK (skipped=1)
2023-01-11T22:55:24.4056543Z 
2023-01-11T22:55:24.4056669Z Generating XML reports...
2023-01-11T22:55:24.4057122Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224655.xml
2023-01-11T22:55:24.4057491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4057663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4058041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4058217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4058237Z 
2023-01-11T22:55:24.4058348Z Running tests...
2023-01-11T22:55:24.4058609Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4058916Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4059222Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4059964Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2023-01-11T22:55:24.4059986Z 
2023-01-11T22:55:24.4060256Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4060395Z Ran 1 test in 1.614s
2023-01-11T22:55:24.4060400Z 
2023-01-11T22:55:24.4060506Z OK (skipped=1)
2023-01-11T22:55:24.4060541Z 
2023-01-11T22:55:24.4060644Z Generating XML reports...
2023-01-11T22:55:24.4061084Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224659.xml
2023-01-11T22:55:24.4061443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4061624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4062015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4062208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4062229Z 
2023-01-11T22:55:24.4062331Z Running tests...
2023-01-11T22:55:24.4062596Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4062890Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4063115Z   test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4063336Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90120
2023-01-11T22:55:24.4063549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90121
2023-01-11T22:55:24.4063977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4064152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4064527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4064715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4065142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4065322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4065677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4065863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4066098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4066345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4066572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4066814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4067216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4067609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4067866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4eyd6phi
2023-01-11T22:55:24.4068119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4eyd6phi/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4068376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg4y41_8n
2023-01-11T22:55:24.4068644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg4y41_8n/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4068746Z ok (7.114s)
2023-01-11T22:55:24.4068765Z 
2023-01-11T22:55:24.4069030Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4069143Z Ran 1 test in 7.114s
2023-01-11T22:55:24.4069163Z 
2023-01-11T22:55:24.4069255Z OK
2023-01-11T22:55:24.4069274Z 
2023-01-11T22:55:24.4069400Z Generating XML reports...
2023-01-11T22:55:24.4069838Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224703.xml
2023-01-11T22:55:24.4070205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4070378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4070763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4070953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4070989Z 
2023-01-11T22:55:24.4071082Z Running tests...
2023-01-11T22:55:24.4071343Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4071657Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4071942Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4072144Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90241
2023-01-11T22:55:24.4072360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90242
2023-01-11T22:55:24.4072727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4072958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4073339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4073528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4073892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4074109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4074475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4074663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4074891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4075139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4075365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4075606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4076005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4076554Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4076951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4077493Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4077747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dnj_hjc
2023-01-11T22:55:24.4077998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dnj_hjc/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4078252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpfxdmsl
2023-01-11T22:55:24.4078520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpfxdmsl/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4078621Z ok (6.620s)
2023-01-11T22:55:24.4078641Z 
2023-01-11T22:55:24.4078909Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4079025Z Ran 1 test in 6.620s
2023-01-11T22:55:24.4079044Z 
2023-01-11T22:55:24.4079136Z OK
2023-01-11T22:55:24.4079155Z 
2023-01-11T22:55:24.4079279Z Generating XML reports...
2023-01-11T22:55:24.4079917Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224713.xml
2023-01-11T22:55:24.4080291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4080474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4080852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4081045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4081065Z 
2023-01-11T22:55:24.4081172Z Running tests...
2023-01-11T22:55:24.4081433Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4081829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4082109Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4082310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90362
2023-01-11T22:55:24.4082526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90363
2023-01-11T22:55:24.4082953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4083136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4083517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4083708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4084069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4084246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4084616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4084783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4085012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4085260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4085487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4085728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4086124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4086672Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4087068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4087608Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4087864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy50tvpb7
2023-01-11T22:55:24.4088132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy50tvpb7/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4088366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp831b_ziu
2023-01-11T22:55:24.4088627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp831b_ziu/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4088730Z ok (6.619s)
2023-01-11T22:55:24.4088750Z 
2023-01-11T22:55:24.4089018Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4089132Z Ran 1 test in 6.619s
2023-01-11T22:55:24.4089151Z 
2023-01-11T22:55:24.4089242Z OK
2023-01-11T22:55:24.4089262Z 
2023-01-11T22:55:24.4089385Z Generating XML reports...
2023-01-11T22:55:24.4089846Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224722.xml
2023-01-11T22:55:24.4090195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4090433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4090812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4091001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4091021Z 
2023-01-11T22:55:24.4091129Z Running tests...
2023-01-11T22:55:24.4091438Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4091759Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4092023Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4092221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90483
2023-01-11T22:55:24.4092440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90484
2023-01-11T22:55:24.4092812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4092986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4093363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4093551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4093918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4094093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4094514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4094682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4094913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4095156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4095383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4095618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4096020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4096412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4096664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi9856lr
2023-01-11T22:55:24.4096930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi9856lr/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4097172Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxo2z3lim
2023-01-11T22:55:24.4097438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxo2z3lim/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4097539Z ok (7.145s)
2023-01-11T22:55:24.4097559Z 
2023-01-11T22:55:24.4097826Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4097937Z Ran 1 test in 7.145s
2023-01-11T22:55:24.4097957Z 
2023-01-11T22:55:24.4098048Z OK
2023-01-11T22:55:24.4098071Z 
2023-01-11T22:55:24.4098195Z Generating XML reports...
2023-01-11T22:55:24.4098654Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224730.xml
2023-01-11T22:55:24.4099002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4099178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4099621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4099811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4099830Z 
2023-01-11T22:55:24.4099940Z Running tests...
2023-01-11T22:55:24.4100201Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4100509Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4100904Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4101131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90604
2023-01-11T22:55:24.4101330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90605
2023-01-11T22:55:24.4101703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4101882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4102258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4102449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4102809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4102987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4103359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4103527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4103753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4103999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4104224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4104459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4104859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4105255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4105511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqu3vk3x2
2023-01-11T22:55:24.4105776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqu3vk3x2/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4106010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplezbj189
2023-01-11T22:55:24.4106278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplezbj189/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4106513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4106739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4106971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4107204Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4107436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4107659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4107881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4108156Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4108385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4108610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4108834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4109055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4109323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4109555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4109780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4109985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4110208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4110431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4110654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4110875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4111095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4111315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4111537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4111742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4111965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4112188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4112407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4112631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4112852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4113077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4113299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4113518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4113722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4113944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4114169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4114390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4114490Z ok (8.724s)
2023-01-11T22:55:24.4114510Z 
2023-01-11T22:55:24.4114786Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4114899Z Ran 1 test in 8.724s
2023-01-11T22:55:24.4114919Z 
2023-01-11T22:55:24.4115011Z OK
2023-01-11T22:55:24.4115031Z 
2023-01-11T22:55:24.4115140Z Generating XML reports...
2023-01-11T22:55:24.4115601Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224740.xml
2023-01-11T22:55:24.4115968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4116145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4116581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4116771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4116792Z 
2023-01-11T22:55:24.4116901Z Running tests...
2023-01-11T22:55:24.4117161Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4117475Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4117779Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4118006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90725
2023-01-11T22:55:24.4118220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90726
2023-01-11T22:55:24.4118594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4118772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4119153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4119345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4120181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4120340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4120720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4120911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4121144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4121374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4121523Z skip: Need at least 4 CUDA devices (3.815s)
2023-01-11T22:55:24.4121544Z 
2023-01-11T22:55:24.4121805Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4121917Z Ran 1 test in 3.815s
2023-01-11T22:55:24.4121936Z 
2023-01-11T22:55:24.4122044Z OK (skipped=1)
2023-01-11T22:55:24.4122063Z 
2023-01-11T22:55:24.4122169Z Generating XML reports...
2023-01-11T22:55:24.4122632Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224751.xml
2023-01-11T22:55:24.4122999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4123174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4123548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4123740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4123759Z 
2023-01-11T22:55:24.4123868Z Running tests...
2023-01-11T22:55:24.4124130Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4124423Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4124698Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4124916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90828
2023-01-11T22:55:24.4125131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90829
2023-01-11T22:55:24.4125498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4125672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4126159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4126347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4126710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4126865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4127309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4127507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4127735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4128280Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4128827Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4129359Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4129893Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4130433Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4130971Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4131201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4131729Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4132264Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4132786Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4133369Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4133943Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4134481Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4134587Z ok (3.925s)
2023-01-11T22:55:24.4134607Z 
2023-01-11T22:55:24.4134884Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4134997Z Ran 1 test in 3.926s
2023-01-11T22:55:24.4135017Z 
2023-01-11T22:55:24.4135090Z OK
2023-01-11T22:55:24.4135128Z 
2023-01-11T22:55:24.4135238Z Generating XML reports...
2023-01-11T22:55:24.4135700Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224757.xml
2023-01-11T22:55:24.4136068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4136244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4136628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4136821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4136840Z 
2023-01-11T22:55:24.4136950Z Running tests...
2023-01-11T22:55:24.4137215Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4137508Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4137802Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4138018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90931
2023-01-11T22:55:24.4138232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90932
2023-01-11T22:55:24.4138597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4138776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4139155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4139342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4139702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4139860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4140232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4140417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4140645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4140951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4141178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4141416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4141820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4142239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4142504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm80tkhq
2023-01-11T22:55:24.4142771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm80tkhq/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4143026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax5eehdd
2023-01-11T22:55:24.4143295Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax5eehdd/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4143396Z ok (7.120s)
2023-01-11T22:55:24.4143417Z 
2023-01-11T22:55:24.4143684Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4143797Z Ran 1 test in 7.120s
2023-01-11T22:55:24.4143816Z 
2023-01-11T22:55:24.4143909Z OK
2023-01-11T22:55:24.4143929Z 
2023-01-11T22:55:24.4144034Z Generating XML reports...
2023-01-11T22:55:24.4144494Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224803.xml
2023-01-11T22:55:24.4144865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4145039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4145413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4145608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4145627Z 
2023-01-11T22:55:24.4145736Z Running tests...
2023-01-11T22:55:24.4145999Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4146288Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4146596Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4146817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91052
2023-01-11T22:55:24.4147034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91053
2023-01-11T22:55:24.4147403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4147575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4147956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4148144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4148505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4148659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4149032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4149218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4149444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4149686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4149972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4150210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4150613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4151008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4151292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqqdwn8j
2023-01-11T22:55:24.4151572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqqdwn8j/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4151821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmz141ewz
2023-01-11T22:55:24.4152083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmz141ewz/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4152188Z ok (7.215s)
2023-01-11T22:55:24.4152208Z 
2023-01-11T22:55:24.4152475Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4152589Z Ran 1 test in 7.216s
2023-01-11T22:55:24.4152608Z 
2023-01-11T22:55:24.4152700Z OK
2023-01-11T22:55:24.4152719Z 
2023-01-11T22:55:24.4152824Z Generating XML reports...
2023-01-11T22:55:24.4153282Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224813.xml
2023-01-11T22:55:24.4153655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4153829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4154205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4154394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4154416Z 
2023-01-11T22:55:24.4154524Z Running tests...
2023-01-11T22:55:24.4154784Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4155094Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4155386Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4155602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91173
2023-01-11T22:55:24.4155816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91174
2023-01-11T22:55:24.4156183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4156357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4156730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4156922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4157284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4157436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4157806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4157995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4158220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4158461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4158688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4158983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4159581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4159977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4160211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jb2we3p
2023-01-11T22:55:24.4160551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jb2we3p/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4160809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpscpymqsa
2023-01-11T22:55:24.4161076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpscpymqsa/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4161308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4161541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4161643Z ok (7.121s)
2023-01-11T22:55:24.4161663Z 
2023-01-11T22:55:24.4161934Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4162048Z Ran 1 test in 7.122s
2023-01-11T22:55:24.4162068Z 
2023-01-11T22:55:24.4162144Z OK
2023-01-11T22:55:24.4162163Z 
2023-01-11T22:55:24.4162287Z Generating XML reports...
2023-01-11T22:55:24.4162749Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224822.xml
2023-01-11T22:55:24.4163118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4163295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4163674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4163867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4163887Z 
2023-01-11T22:55:24.4163995Z Running tests...
2023-01-11T22:55:24.4164239Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4164547Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4164857Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4165076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91294
2023-01-11T22:55:24.4165293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91295
2023-01-11T22:55:24.4165660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4165835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4166211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4166397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4166740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4166912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4167289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4167474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4167700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4167949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4168246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4168486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4168869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4169263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4169567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0zjuiit
2023-01-11T22:55:24.4169843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0zjuiit/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4170094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_k9p5l0
2023-01-11T22:55:24.4170358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_k9p5l0/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4170594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4170828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4170929Z ok (7.120s)
2023-01-11T22:55:24.4170949Z 
2023-01-11T22:55:24.4171202Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4171314Z Ran 1 test in 7.120s
2023-01-11T22:55:24.4171333Z 
2023-01-11T22:55:24.4171425Z OK
2023-01-11T22:55:24.4171445Z 
2023-01-11T22:55:24.4171571Z Generating XML reports...
2023-01-11T22:55:24.4172028Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224832.xml
2023-01-11T22:55:24.4172397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4172571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4172957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4173146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4173168Z 
2023-01-11T22:55:24.4173258Z Running tests...
2023-01-11T22:55:24.4173519Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4173829Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4174106Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4174323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91415
2023-01-11T22:55:24.4174538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91416
2023-01-11T22:55:24.4174906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4175082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4175440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4175629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4175987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4176162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4176533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4176719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4176946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4177163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4177353Z skip: Need at least 4 CUDA devices (3.906s)
2023-01-11T22:55:24.4177394Z 
2023-01-11T22:55:24.4177642Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4177756Z Ran 1 test in 3.906s
2023-01-11T22:55:24.4177775Z 
2023-01-11T22:55:24.4177883Z OK (skipped=1)
2023-01-11T22:55:24.4177902Z 
2023-01-11T22:55:24.4178026Z Generating XML reports...
2023-01-11T22:55:24.4178522Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224841.xml
2023-01-11T22:55:24.4178900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4179076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4179453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4179629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4179668Z 
2023-01-11T22:55:24.4179759Z Running tests...
2023-01-11T22:55:24.4180024Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4180332Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4180608Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4180827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91518
2023-01-11T22:55:24.4181043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91519
2023-01-11T22:55:24.4181411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4181586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4181947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4182137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4182498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4182669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4183047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4183234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4183516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4183745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4183877Z skip: Need at least 8 CUDA devices (3.818s)
2023-01-11T22:55:24.4183919Z 
2023-01-11T22:55:24.4184168Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4184280Z Ran 1 test in 3.818s
2023-01-11T22:55:24.4184300Z 
2023-01-11T22:55:24.4184407Z OK (skipped=1)
2023-01-11T22:55:24.4184425Z 
2023-01-11T22:55:24.4184548Z Generating XML reports...
2023-01-11T22:55:24.4184999Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224847.xml
2023-01-11T22:55:24.4185366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4185538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4185911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4186080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4186154Z 
2023-01-11T22:55:24.4186268Z Running tests...
2023-01-11T22:55:24.4186529Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4186836Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4187128Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4187346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91621
2023-01-11T22:55:24.4187617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91622
2023-01-11T22:55:24.4187993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4188148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4188524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4188713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4189076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4189246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4189614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4189800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4190030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4190272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4190481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4190722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4191123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4191514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4191617Z ok (5.532s)
2023-01-11T22:55:24.4191636Z 
2023-01-11T22:55:24.4191898Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4192012Z Ran 1 test in 5.533s
2023-01-11T22:55:24.4192035Z 
2023-01-11T22:55:24.4192129Z OK
2023-01-11T22:55:24.4192148Z 
2023-01-11T22:55:24.4192269Z Generating XML reports...
2023-01-11T22:55:24.4192707Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224854.xml
2023-01-11T22:55:24.4193071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4193250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4193623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4193810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4193830Z 
2023-01-11T22:55:24.4193937Z Running tests...
2023-01-11T22:55:24.4194244Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4194560Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4194850Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4195070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91730
2023-01-11T22:55:24.4195285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91731
2023-01-11T22:55:24.4195718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4195892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4196264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4196454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4196859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4197039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4197395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4197583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4197815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4198042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4198190Z skip: Need at least 4 CUDA devices (3.921s)
2023-01-11T22:55:24.4198211Z 
2023-01-11T22:55:24.4198474Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4198587Z Ran 1 test in 3.921s
2023-01-11T22:55:24.4198607Z 
2023-01-11T22:55:24.4198714Z OK (skipped=1)
2023-01-11T22:55:24.4198733Z 
2023-01-11T22:55:24.4198842Z Generating XML reports...
2023-01-11T22:55:24.4199298Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224901.xml
2023-01-11T22:55:24.4199887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4200065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4200449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4200637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4200657Z 
2023-01-11T22:55:24.4200766Z Running tests...
2023-01-11T22:55:24.4201026Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4201334Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4201629Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4201848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91833
2023-01-11T22:55:24.4202062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91834
2023-01-11T22:55:24.4202426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4202602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4202974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4203164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4203521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4203680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4204052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4204239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4204466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4204805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4205026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4205270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4205676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4206129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4206376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_q3kqn1q
2023-01-11T22:55:24.4206645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_q3kqn1q/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4206899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknw805s8
2023-01-11T22:55:24.4207170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknw805s8/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4207398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4207630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4207757Z ok (7.134s)
2023-01-11T22:55:24.4207768Z 
2023-01-11T22:55:24.4208030Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4208145Z Ran 1 test in 7.134s
2023-01-11T22:55:24.4208166Z 
2023-01-11T22:55:24.4208249Z OK
2023-01-11T22:55:24.4208270Z 
2023-01-11T22:55:24.4208378Z Generating XML reports...
2023-01-11T22:55:24.4208840Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224908.xml
2023-01-11T22:55:24.4209233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4209421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4209809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4210000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4210019Z 
2023-01-11T22:55:24.4210142Z Running tests...
2023-01-11T22:55:24.4210396Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4210702Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4211002Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4211218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91954
2023-01-11T22:55:24.4211433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91955
2023-01-11T22:55:24.4211806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4211979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4212355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4212542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4212887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4213058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4213426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4213611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4213836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4214140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4214368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4214605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4215056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4215442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4215697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr27iwc6c
2023-01-11T22:55:24.4215962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr27iwc6c/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4216217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqwokuer
2023-01-11T22:55:24.4216482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqwokuer/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4216715Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4216940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4217041Z ok (7.113s)
2023-01-11T22:55:24.4217061Z 
2023-01-11T22:55:24.4217312Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4217425Z Ran 1 test in 7.113s
2023-01-11T22:55:24.4217444Z 
2023-01-11T22:55:24.4217536Z OK
2023-01-11T22:55:24.4217555Z 
2023-01-11T22:55:24.4217676Z Generating XML reports...
2023-01-11T22:55:24.4218134Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224917.xml
2023-01-11T22:55:24.4218507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4218681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4219054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4219241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4219261Z 
2023-01-11T22:55:24.4219352Z Running tests...
2023-01-11T22:55:24.4219615Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4219925Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4220204Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4220424Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92075
2023-01-11T22:55:24.4220644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92076
2023-01-11T22:55:24.4221013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4221186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4221541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4221728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4222092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4222263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4222634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4222821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4223107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4223336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4223440Z ok (21.828s)
2023-01-11T22:55:24.4223460Z 
2023-01-11T22:55:24.4223707Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4223822Z Ran 1 test in 21.828s
2023-01-11T22:55:24.4223841Z 
2023-01-11T22:55:24.4223933Z OK
2023-01-11T22:55:24.4223998Z 
2023-01-11T22:55:24.4224128Z Generating XML reports...
2023-01-11T22:55:24.4224588Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224926.xml
2023-01-11T22:55:24.4224953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4225127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4225502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4225670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4225709Z 
2023-01-11T22:55:24.4225800Z Running tests...
2023-01-11T22:55:24.4226057Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4226365Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4226550Z   test_no_grad (__main__.DistributedDataParallelTest)
2023-01-11T22:55:24.4226800Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4227017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92196
2023-01-11T22:55:24.4227232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92197
2023-01-11T22:55:24.4227600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4227758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4228137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4228325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4228688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4228856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4229228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4229414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4229643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4229872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4230093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4230330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4230727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4231118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4231372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpno62rcsr
2023-01-11T22:55:24.4231638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpno62rcsr/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4231949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp957txy4y
2023-01-11T22:55:24.4232213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp957txy4y/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4232298Z ok (7.127s)
2023-01-11T22:55:24.4232318Z 
2023-01-11T22:55:24.4232586Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4232699Z Ran 1 test in 7.127s
2023-01-11T22:55:24.4232718Z 
2023-01-11T22:55:24.4232812Z OK
2023-01-11T22:55:24.4232831Z 
2023-01-11T22:55:24.4232956Z Generating XML reports...
2023-01-11T22:55:24.4233456Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224951.xml
2023-01-11T22:55:24.4233834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4234008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4234390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4234560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4234579Z 
2023-01-11T22:55:24.4234688Z Running tests...
2023-01-11T22:55:24.4234946Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4235256Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4235536Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4235760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92313
2023-01-11T22:55:24.4235976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92314
2023-01-11T22:55:24.4236342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4236503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4236881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4237069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4237432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4237606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4237977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4238163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4238389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4238615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4238840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4239083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4239773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4240189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4240448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcagbkukb
2023-01-11T22:55:24.4240718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcagbkukb/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4240973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocu5lgnh
2023-01-11T22:55:24.4241238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocu5lgnh/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4241427Z ok (6.635s)
2023-01-11T22:55:24.4241467Z 
2023-01-11T22:55:24.4241716Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4241830Z Ran 1 test in 6.635s
2023-01-11T22:55:24.4241850Z 
2023-01-11T22:55:24.4241942Z OK
2023-01-11T22:55:24.4241962Z 
2023-01-11T22:55:24.4242085Z Generating XML reports...
2023-01-11T22:55:24.4242592Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225000.xml
2023-01-11T22:55:24.4242973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4243147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4243523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4243701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4243720Z 
2023-01-11T22:55:24.4243830Z Running tests...
2023-01-11T22:55:24.4244090Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4244397Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4244658Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4244878Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92430
2023-01-11T22:55:24.4245096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92431
2023-01-11T22:55:24.4245463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4245618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4245991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4246181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4246543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4246713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4247080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4247271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4247497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4247741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4247945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4248189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4248585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4248977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4249077Z ok (3.911s)
2023-01-11T22:55:24.4249097Z 
2023-01-11T22:55:24.4249361Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4249476Z Ran 1 test in 3.911s
2023-01-11T22:55:24.4249495Z 
2023-01-11T22:55:24.4249588Z OK
2023-01-11T22:55:24.4249607Z 
2023-01-11T22:55:24.4249729Z Generating XML reports...
2023-01-11T22:55:24.4250165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225009.xml
2023-01-11T22:55:24.4250532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4250764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4251143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4251333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4251353Z 
2023-01-11T22:55:24.4251462Z Running tests...
2023-01-11T22:55:24.4251777Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4252100Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4252361Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4252579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92537
2023-01-11T22:55:24.4252799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92538
2023-01-11T22:55:24.4253167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4253339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4253711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4253900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4254262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4254434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4254787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4254972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4255204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4255443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4255669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4255905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4256306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4256702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4257247Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4257778Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4258019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8oqswmh
2023-01-11T22:55:24.4258285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8oqswmh/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4258533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_y5exeyc
2023-01-11T22:55:24.4258798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_y5exeyc/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4259386Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4259960Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4260498Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4261038Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4261569Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4262088Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4262620Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4263147Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4263670Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4264190Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4264725Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4265293Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4265867Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4266399Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4266507Z ok (6.737s)
2023-01-11T22:55:24.4266527Z 
2023-01-11T22:55:24.4266798Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4266910Z Ran 1 test in 6.737s
2023-01-11T22:55:24.4266930Z 
2023-01-11T22:55:24.4267023Z OK
2023-01-11T22:55:24.4267042Z 
2023-01-11T22:55:24.4267166Z Generating XML reports...
2023-01-11T22:55:24.4267629Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225015.xml
2023-01-11T22:55:24.4267983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4268161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4268536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4268733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4268753Z 
2023-01-11T22:55:24.4268860Z Running tests...
2023-01-11T22:55:24.4269123Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4269432Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4269724Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4269930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92658
2023-01-11T22:55:24.4270144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92659
2023-01-11T22:55:24.4270513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4270687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4271071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4271261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4271622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4271794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4272164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4272333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4272565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4272805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4273030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4273319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4273721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4274298Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4274706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4275241Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4275499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyro2w36
2023-01-11T22:55:24.4275765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyro2w36/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4275997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1bs_yjx
2023-01-11T22:55:24.4276262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1bs_yjx/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4276795Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4277335Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4277864Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4278397Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4278929Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4279665Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4280200Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4280855Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4281385Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4281904Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4282441Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4282961Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:55:24.4283496Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4284021Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:55:24.4284124Z ok (6.623s)
2023-01-11T22:55:24.4284145Z 
2023-01-11T22:55:24.4284421Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4284533Z Ran 1 test in 6.623s
2023-01-11T22:55:24.4284557Z 
2023-01-11T22:55:24.4284650Z OK
2023-01-11T22:55:24.4284670Z 
2023-01-11T22:55:24.4284795Z Generating XML reports...
2023-01-11T22:55:24.4285253Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225024.xml
2023-01-11T22:55:24.4285624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4285781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4286162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4286353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4286372Z 
2023-01-11T22:55:24.4286481Z Running tests...
2023-01-11T22:55:24.4286744Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4287049Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4287381Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4287600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92779
2023-01-11T22:55:24.4287797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92780
2023-01-11T22:55:24.4288170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4288387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4288773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4288961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4289319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4289492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4289862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4290047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4290258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4290506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4290732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4290967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4291366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4291761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4292016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kax2b8b
2023-01-11T22:55:24.4292284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kax2b8b/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4292532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s2m8c7u
2023-01-11T22:55:24.4292781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s2m8c7u/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4293017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4293247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4293348Z ok (8.138s)
2023-01-11T22:55:24.4293369Z 
2023-01-11T22:55:24.4293636Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4293750Z Ran 1 test in 8.139s
2023-01-11T22:55:24.4293770Z 
2023-01-11T22:55:24.4293862Z OK
2023-01-11T22:55:24.4293882Z 
2023-01-11T22:55:24.4294006Z Generating XML reports...
2023-01-11T22:55:24.4294491Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225033.xml
2023-01-11T22:55:24.4294864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4295040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4295414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4295603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4295623Z 
2023-01-11T22:55:24.4295732Z Running tests...
2023-01-11T22:55:24.4295992Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4296368Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4296650Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4296849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92900
2023-01-11T22:55:24.4297064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92901
2023-01-11T22:55:24.4297479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4297663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4298038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4298229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4298591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4298761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4299108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4299296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4299523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4299769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4299992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4300228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4300624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4301021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4301275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovmpe_q4
2023-01-11T22:55:24.4301523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovmpe_q4/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4301775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp20_qvm9f
2023-01-11T22:55:24.4302036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp20_qvm9f/_remote_module_non_scriptable.py
2023-01-11T22:55:24.4302273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4302504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:55:24.4302608Z ok (7.312s)
2023-01-11T22:55:24.4302628Z 
2023-01-11T22:55:24.4302895Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4303006Z Ran 1 test in 7.312s
2023-01-11T22:55:24.4303026Z 
2023-01-11T22:55:24.4303101Z OK
2023-01-11T22:55:24.4303138Z 
2023-01-11T22:55:24.4303246Z Generating XML reports...
2023-01-11T22:55:24.4303703Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225044.xml
2023-01-11T22:55:24.4304072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4304247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4304625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4304816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4304886Z 
2023-01-11T22:55:24.4305002Z Running tests...
2023-01-11T22:55:24.4305269Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4305558Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4305825Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4306040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93021
2023-01-11T22:55:24.4306311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93022
2023-01-11T22:55:24.4306535Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93023
2023-01-11T22:55:24.4306906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4307079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4307454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4307627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4307989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4308160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4308527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4308719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4309077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4309248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4309612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4309799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4310009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4310227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:55:24.4310450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4310599Z skip: Need at least 3 CUDA devices (3.928s)
2023-01-11T22:55:24.4310622Z 
2023-01-11T22:55:24.4310888Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4311001Z Ran 1 test in 3.928s
2023-01-11T22:55:24.4311021Z 
2023-01-11T22:55:24.4311127Z OK (skipped=1)
2023-01-11T22:55:24.4311146Z 
2023-01-11T22:55:24.4311268Z Generating XML reports...
2023-01-11T22:55:24.4311678Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225053.xml
2023-01-11T22:55:24.4312050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4312223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4312597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4312787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4312807Z 
2023-01-11T22:55:24.4312917Z Running tests...
2023-01-11T22:55:24.4313180Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4313488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4313753Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4313954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93158
2023-01-11T22:55:24.4314229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93159
2023-01-11T22:55:24.4314443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93160
2023-01-11T22:55:24.4314814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4314989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4315406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4315601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4315967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4316121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4316490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4316661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4317034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4317222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4317593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4317777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4318001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:55:24.4318225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4318431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4318582Z skip: Need at least 3 CUDA devices (4.026s)
2023-01-11T22:55:24.4318602Z 
2023-01-11T22:55:24.4318865Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4318978Z Ran 1 test in 4.027s
2023-01-11T22:55:24.4318997Z 
2023-01-11T22:55:24.4319104Z OK (skipped=1)
2023-01-11T22:55:24.4319124Z 
2023-01-11T22:55:24.4319247Z Generating XML reports...
2023-01-11T22:55:24.4319902Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225100.xml
2023-01-11T22:55:24.4320285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4320456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4320811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4321004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4321024Z 
2023-01-11T22:55:24.4321133Z Running tests...
2023-01-11T22:55:24.4321394Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4321703Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4322031Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.000s)
2023-01-11T22:55:24.4322051Z 
2023-01-11T22:55:24.4322310Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4322422Z Ran 1 test in 0.001s
2023-01-11T22:55:24.4322441Z 
2023-01-11T22:55:24.4322531Z OK (skipped=1)
2023-01-11T22:55:24.4322571Z 
2023-01-11T22:55:24.4322676Z Generating XML reports...
2023-01-11T22:55:24.4323106Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225106.xml
2023-01-11T22:55:24.4323562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4323736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4324112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4324301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4324324Z 
2023-01-11T22:55:24.4324433Z Running tests...
2023-01-11T22:55:24.4324750Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4325054Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4325321Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4325538Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93328
2023-01-11T22:55:24.4325757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93329
2023-01-11T22:55:24.4325969Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93330
2023-01-11T22:55:24.4326336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4326509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4326884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4327052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4327418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4327589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4327955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4328142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4328501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4328674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4329039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4329226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4329436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4329662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4329886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:55:24.4330040Z skip: Need at least 3 CUDA devices (4.020s)
2023-01-11T22:55:24.4330060Z 
2023-01-11T22:55:24.4330329Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4330441Z Ran 1 test in 4.021s
2023-01-11T22:55:24.4330460Z 
2023-01-11T22:55:24.4330568Z OK (skipped=1)
2023-01-11T22:55:24.4330586Z 
2023-01-11T22:55:24.4330709Z Generating XML reports...
2023-01-11T22:55:24.4331118Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225108.xml
2023-01-11T22:55:24.4331485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4331659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4332031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4332279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4332299Z 
2023-01-11T22:55:24.4332408Z Running tests...
2023-01-11T22:55:24.4332671Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4332983Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4333251Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4333493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93465
2023-01-11T22:55:24.4333715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93466
2023-01-11T22:55:24.4333928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93467
2023-01-11T22:55:24.4334295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4334470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4334848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4335039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4335399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4335573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4335925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4336109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4336466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4336636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4337013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4337197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4337423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4337648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:55:24.4337857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4337960Z ok (4.011s)
2023-01-11T22:55:24.4337980Z 
2023-01-11T22:55:24.4338243Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4338355Z Ran 1 test in 4.011s
2023-01-11T22:55:24.4338374Z 
2023-01-11T22:55:24.4338469Z OK
2023-01-11T22:55:24.4338488Z 
2023-01-11T22:55:24.4338610Z Generating XML reports...
2023-01-11T22:55:24.4339034Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225115.xml
2023-01-11T22:55:24.4339400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4339555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4339931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4340119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4340141Z 
2023-01-11T22:55:24.4340251Z Running tests...
2023-01-11T22:55:24.4340515Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4340824Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4341085Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4341361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93602
2023-01-11T22:55:24.4341576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93603
2023-01-11T22:55:24.4341771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93604
2023-01-11T22:55:24.4342141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4342314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4342731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4342929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4343297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4343470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4343844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4344014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4344372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4344543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4344914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4345098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4345322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:55:24.4345541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4345765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4345866Z ok (3.922s)
2023-01-11T22:55:24.4345886Z 
2023-01-11T22:55:24.4346133Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4346247Z Ran 1 test in 3.922s
2023-01-11T22:55:24.4346266Z 
2023-01-11T22:55:24.4346359Z OK
2023-01-11T22:55:24.4346378Z 
2023-01-11T22:55:24.4346501Z Generating XML reports...
2023-01-11T22:55:24.4346928Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225121.xml
2023-01-11T22:55:24.4347292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4347465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4347834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4348006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4348043Z 
2023-01-11T22:55:24.4348135Z Running tests...
2023-01-11T22:55:24.4348395Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4348705Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4348966Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4349185Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93739
2023-01-11T22:55:24.4349400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93740
2023-01-11T22:55:24.4349612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93741
2023-01-11T22:55:24.4349979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4350190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4350571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4350761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4351122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4351295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4351714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4351907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4352271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4352424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4352793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4352976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4353205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:55:24.4353428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4353656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4353758Z ok (3.930s)
2023-01-11T22:55:24.4353778Z 
2023-01-11T22:55:24.4354040Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4354135Z Ran 1 test in 3.931s
2023-01-11T22:55:24.4354171Z 
2023-01-11T22:55:24.4354247Z OK
2023-01-11T22:55:24.4354266Z 
2023-01-11T22:55:24.4354389Z Generating XML reports...
2023-01-11T22:55:24.4354822Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225127.xml
2023-01-11T22:55:24.4355189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4355362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4355728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4355917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4355937Z 
2023-01-11T22:55:24.4356045Z Running tests...
2023-01-11T22:55:24.4356289Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4356601Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4356860Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s)
2023-01-11T22:55:24.4356884Z 
2023-01-11T22:55:24.4357140Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4357252Z Ran 1 test in 0.001s
2023-01-11T22:55:24.4357271Z 
2023-01-11T22:55:24.4357378Z OK (skipped=1)
2023-01-11T22:55:24.4357397Z 
2023-01-11T22:55:24.4357519Z Generating XML reports...
2023-01-11T22:55:24.4357938Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225133.xml
2023-01-11T22:55:24.4358308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4358463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4358835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4359022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4359092Z 
2023-01-11T22:55:24.4359206Z Running tests...
2023-01-11T22:55:24.4359743Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4360065Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4360313Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4360521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93909
2023-01-11T22:55:24.4360788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93910
2023-01-11T22:55:24.4361013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93911
2023-01-11T22:55:24.4361385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4361559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4361930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4362124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4362482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4362652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4363017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4363191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4363548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4363717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4364080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4364267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4364493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4364714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4364937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:55:24.4365069Z skip: Need at least 3 CUDA devices (3.926s)
2023-01-11T22:55:24.4365109Z 
2023-01-11T22:55:24.4365355Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4365468Z Ran 1 test in 3.927s
2023-01-11T22:55:24.4365488Z 
2023-01-11T22:55:24.4365595Z OK (skipped=1)
2023-01-11T22:55:24.4365614Z 
2023-01-11T22:55:24.4365736Z Generating XML reports...
2023-01-11T22:55:24.4366165Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225136.xml
2023-01-11T22:55:24.4366535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4366708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4367081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4367251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4367271Z 
2023-01-11T22:55:24.4367383Z Running tests...
2023-01-11T22:55:24.4367644Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4367954Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4368277Z   test_all_to_all_single (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4368494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94046
2023-01-11T22:55:24.4368937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4369110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4369464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4369652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4369987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4370241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4370644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4370749Z ok (5.224s)
2023-01-11T22:55:24.4370773Z 
2023-01-11T22:55:24.4371035Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4371147Z Ran 1 test in 5.224s
2023-01-11T22:55:24.4371167Z 
2023-01-11T22:55:24.4371259Z OK
2023-01-11T22:55:24.4371279Z 
2023-01-11T22:55:24.4371384Z Generating XML reports...
2023-01-11T22:55:24.4371931Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225142.xml
2023-01-11T22:55:24.4372300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4372474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4372850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4373039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4373059Z 
2023-01-11T22:55:24.4373170Z Running tests...
2023-01-11T22:55:24.4373428Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4373720Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4374045Z   test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4374262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94121
2023-01-11T22:55:24.4374631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4374804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4375181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4375372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4375597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4375843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4376223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4376326Z ok (5.318s)
2023-01-11T22:55:24.4376346Z 
2023-01-11T22:55:24.4376607Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4376718Z Ran 1 test in 5.319s
2023-01-11T22:55:24.4376742Z 
2023-01-11T22:55:24.4376835Z OK
2023-01-11T22:55:24.4376854Z 
2023-01-11T22:55:24.4376978Z Generating XML reports...
2023-01-11T22:55:24.4377526Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225149.xml
2023-01-11T22:55:24.4377892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4378125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4378487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4378679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4378699Z 
2023-01-11T22:55:24.4378810Z Running tests...
2023-01-11T22:55:24.4379072Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4379424Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4379768Z   test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4379985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94196
2023-01-11T22:55:24.4380354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4380514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4380891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4381078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4381304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4381548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4381949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4382678Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:55:24.4382796Z   warnings.warn(
2023-01-11T22:55:24.4382897Z ok (5.331s)
2023-01-11T22:55:24.4382917Z 
2023-01-11T22:55:24.4383163Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4383275Z Ran 1 test in 5.331s
2023-01-11T22:55:24.4383294Z 
2023-01-11T22:55:24.4383387Z OK
2023-01-11T22:55:24.4383406Z 
2023-01-11T22:55:24.4383528Z Generating XML reports...
2023-01-11T22:55:24.4384076Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225157.xml
2023-01-11T22:55:24.4384446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4384622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4384999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4385190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4385210Z 
2023-01-11T22:55:24.4385300Z Running tests...
2023-01-11T22:55:24.4385561Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4385871Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4386190Z   test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4386408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94271
2023-01-11T22:55:24.4386775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4386949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4387323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4387555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4387785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4388032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4388434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4388536Z ok (5.333s)
2023-01-11T22:55:24.4388606Z 
2023-01-11T22:55:24.4388877Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4388991Z Ran 1 test in 5.333s
2023-01-11T22:55:24.4389011Z 
2023-01-11T22:55:24.4389103Z OK
2023-01-11T22:55:24.4389123Z 
2023-01-11T22:55:24.4389249Z Generating XML reports...
2023-01-11T22:55:24.4389773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225205.xml
2023-01-11T22:55:24.4390144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4390316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4390690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4390878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4390898Z 
2023-01-11T22:55:24.4391009Z Running tests...
2023-01-11T22:55:24.4391271Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4391578Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4391907Z   test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4392108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94346
2023-01-11T22:55:24.4392477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4392649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4393024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4393212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4393442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4393681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4394079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4394168Z ok (5.325s)
2023-01-11T22:55:24.4394205Z 
2023-01-11T22:55:24.4394498Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4394612Z Ran 1 test in 5.325s
2023-01-11T22:55:24.4394633Z 
2023-01-11T22:55:24.4394726Z OK
2023-01-11T22:55:24.4394745Z 
2023-01-11T22:55:24.4394868Z Generating XML reports...
2023-01-11T22:55:24.4395414Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225212.xml
2023-01-11T22:55:24.4395784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4395958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4396333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4396505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4396579Z 
2023-01-11T22:55:24.4396694Z Running tests...
2023-01-11T22:55:24.4396958Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4397266Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4397528Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s)
2023-01-11T22:55:24.4397548Z 
2023-01-11T22:55:24.4397806Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4397917Z Ran 1 test in 0.001s
2023-01-11T22:55:24.4397982Z 
2023-01-11T22:55:24.4398097Z OK (skipped=1)
2023-01-11T22:55:24.4398116Z 
2023-01-11T22:55:24.4398240Z Generating XML reports...
2023-01-11T22:55:24.4398674Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20230111225220.xml
2023-01-11T22:55:24.4399040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4399219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4399824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4400015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4400043Z 
2023-01-11T22:55:24.4400141Z Running tests...
2023-01-11T22:55:24.4400408Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4400721Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4400957Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4401175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94454
2023-01-11T22:55:24.4401392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94455
2023-01-11T22:55:24.4401763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4401938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4402313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4402502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4402864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4403038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4403397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4403583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4403809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4404056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4404279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4404520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4404919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4405320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4405405Z ok (5.425s)
2023-01-11T22:55:24.4405443Z 
2023-01-11T22:55:24.4405691Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4405802Z Ran 1 test in 5.425s
2023-01-11T22:55:24.4405821Z 
2023-01-11T22:55:24.4405995Z OK
2023-01-11T22:55:24.4406015Z 
2023-01-11T22:55:24.4406141Z Generating XML reports...
2023-01-11T22:55:24.4406574Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225222.xml
2023-01-11T22:55:24.4406938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4407113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4407548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4407728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4407747Z 
2023-01-11T22:55:24.4407857Z Running tests...
2023-01-11T22:55:24.4408122Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4408432Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4408687Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4408902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94563
2023-01-11T22:55:24.4409114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94564
2023-01-11T22:55:24.4409483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4409637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4410015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4410203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4410562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4410733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4411107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4411294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4411520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4411764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4411979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4412234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4412625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4413006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4413111Z ok (6.647s)
2023-01-11T22:55:24.4413130Z 
2023-01-11T22:55:24.4413394Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4413506Z Ran 1 test in 6.647s
2023-01-11T22:55:24.4413525Z 
2023-01-11T22:55:24.4413617Z OK
2023-01-11T22:55:24.4413636Z 
2023-01-11T22:55:24.4413742Z Generating XML reports...
2023-01-11T22:55:24.4414169Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225230.xml
2023-01-11T22:55:24.4414538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4414713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4415100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4415276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4415360Z 
2023-01-11T22:55:24.4415478Z Running tests...
2023-01-11T22:55:24.4415743Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4416052Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4416276Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4416492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94680
2023-01-11T22:55:24.4416751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94681
2023-01-11T22:55:24.4417129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4417302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4417676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4417867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4418226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4418379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4418750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4418943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4419170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4419412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4419632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4419876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4420272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4420664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4420750Z ok (6.633s)
2023-01-11T22:55:24.4420769Z 
2023-01-11T22:55:24.4421031Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4421147Z Ran 1 test in 6.633s
2023-01-11T22:55:24.4421166Z 
2023-01-11T22:55:24.4421259Z OK
2023-01-11T22:55:24.4421278Z 
2023-01-11T22:55:24.4421400Z Generating XML reports...
2023-01-11T22:55:24.4421831Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225239.xml
2023-01-11T22:55:24.4422199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4422377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4422751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4422922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4422942Z 
2023-01-11T22:55:24.4423050Z Running tests...
2023-01-11T22:55:24.4423311Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4423624Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4423866Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4424083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94797
2023-01-11T22:55:24.4424294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94798
2023-01-11T22:55:24.4424721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4424878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4425250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4425438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4425841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4426019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4426397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4426582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4426814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4427055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4427259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4427495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4427896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4428288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4428390Z ok (6.631s)
2023-01-11T22:55:24.4428409Z 
2023-01-11T22:55:24.4428672Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4428784Z Ran 1 test in 6.631s
2023-01-11T22:55:24.4428808Z 
2023-01-11T22:55:24.4428898Z OK
2023-01-11T22:55:24.4428918Z 
2023-01-11T22:55:24.4429024Z Generating XML reports...
2023-01-11T22:55:24.4429450Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225248.xml
2023-01-11T22:55:24.4429814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4429987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4430368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4430559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4430579Z 
2023-01-11T22:55:24.4430686Z Running tests...
2023-01-11T22:55:24.4430948Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4431257Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4431473Z   test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4431689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94914
2023-01-11T22:55:24.4431904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94915
2023-01-11T22:55:24.4432265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4432438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4432814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4433003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4433363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4433574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4433948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4434133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4434361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4434585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4434885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4435137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4435535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4435927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4436016Z ok (6.716s)
2023-01-11T22:55:24.4436037Z 
2023-01-11T22:55:24.4436298Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4436408Z Ran 1 test in 6.717s
2023-01-11T22:55:24.4436428Z 
2023-01-11T22:55:24.4436522Z OK
2023-01-11T22:55:24.4436541Z 
2023-01-11T22:55:24.4436663Z Generating XML reports...
2023-01-11T22:55:24.4437090Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225257.xml
2023-01-11T22:55:24.4437459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4437632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4437990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4438185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4438205Z 
2023-01-11T22:55:24.4438314Z Running tests...
2023-01-11T22:55:24.4438578Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4438885Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4439125Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4439343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95031
2023-01-11T22:55:24.4439781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95032
2023-01-11T22:55:24.4440158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4440310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4440698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4440878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4441236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4441406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4441774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4441968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4442196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4442419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4442641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4442961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4443361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4443753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4443857Z ok (6.615s)
2023-01-11T22:55:24.4443877Z 
2023-01-11T22:55:24.4444191Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4444312Z Ran 1 test in 6.615s
2023-01-11T22:55:24.4444332Z 
2023-01-11T22:55:24.4444426Z OK
2023-01-11T22:55:24.4444445Z 
2023-01-11T22:55:24.4444551Z Generating XML reports...
2023-01-11T22:55:24.4444984Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225306.xml
2023-01-11T22:55:24.4445349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4445526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4445897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4446084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4446104Z 
2023-01-11T22:55:24.4446211Z Running tests...
2023-01-11T22:55:24.4446474Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4446780Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4447003Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4447221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95148
2023-01-11T22:55:24.4447435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95149
2023-01-11T22:55:24.4447807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4447980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4448351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4448540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4448902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4449053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4449428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4449614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4449843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4450085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4450309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4450546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4450943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4451335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4451419Z ok (6.724s)
2023-01-11T22:55:24.4451439Z 
2023-01-11T22:55:24.4451701Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4451813Z Ran 1 test in 6.724s
2023-01-11T22:55:24.4451884Z 
2023-01-11T22:55:24.4451986Z OK
2023-01-11T22:55:24.4452005Z 
2023-01-11T22:55:24.4452131Z Generating XML reports...
2023-01-11T22:55:24.4452564Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225315.xml
2023-01-11T22:55:24.4452928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4453102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4453531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4453733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4453754Z 
2023-01-11T22:55:24.4453864Z Running tests...
2023-01-11T22:55:24.4454128Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4454439Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4454682Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4454898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95265
2023-01-11T22:55:24.4455110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95266
2023-01-11T22:55:24.4455476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4455634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4456004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4456191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4456549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4456726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4457092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4457276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4457500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4457725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4457949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4458189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4458587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4458984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4459085Z ok (5.418s)
2023-01-11T22:55:24.4459104Z 
2023-01-11T22:55:24.4459367Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4459480Z Ran 1 test in 5.418s
2023-01-11T22:55:24.4459500Z 
2023-01-11T22:55:24.4459593Z OK
2023-01-11T22:55:24.4459612Z 
2023-01-11T22:55:24.4459718Z Generating XML reports...
2023-01-11T22:55:24.4460146Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225324.xml
2023-01-11T22:55:24.4460510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4460683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4461059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4461308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4461327Z 
2023-01-11T22:55:24.4461435Z Running tests...
2023-01-11T22:55:24.4461699Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4461990Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4462229Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4462492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95374
2023-01-11T22:55:24.4462719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95375
2023-01-11T22:55:24.4463089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4463263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4463627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4463804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4464177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4464347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4464716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4464906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4465131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4465372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4465600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4465845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4466241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4466631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4466716Z ok (6.639s)
2023-01-11T22:55:24.4466736Z 
2023-01-11T22:55:24.4466999Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4467111Z Ran 1 test in 6.640s
2023-01-11T22:55:24.4467130Z 
2023-01-11T22:55:24.4467221Z OK
2023-01-11T22:55:24.4467241Z 
2023-01-11T22:55:24.4467363Z Generating XML reports...
2023-01-11T22:55:24.4467789Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225332.xml
2023-01-11T22:55:24.4468155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4468329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4468681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4468868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4468888Z 
2023-01-11T22:55:24.4468995Z Running tests...
2023-01-11T22:55:24.4469259Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4469569Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4469811Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4470028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95495
2023-01-11T22:55:24.4470241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95496
2023-01-11T22:55:24.4470653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4470827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4471202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4471389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4471790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4471968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4472345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4472530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4472759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4472983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4473205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4473442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4473841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4474236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4474337Z ok (11.137s)
2023-01-11T22:55:24.4474357Z 
2023-01-11T22:55:24.4474619Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4474737Z Ran 1 test in 11.137s
2023-01-11T22:55:24.4474757Z 
2023-01-11T22:55:24.4474850Z OK
2023-01-11T22:55:24.4474869Z 
2023-01-11T22:55:24.4474975Z Generating XML reports...
2023-01-11T22:55:24.4475405Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225341.xml
2023-01-11T22:55:24.4475767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4475941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4476319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4476509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4476528Z 
2023-01-11T22:55:24.4476637Z Running tests...
2023-01-11T22:55:24.4476904Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4477195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4477455Z   test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4477668Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95616
2023-01-11T22:55:24.4477881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95617
2023-01-11T22:55:24.4478245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4478423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4478798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4478984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4479336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4479775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4480157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4480342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4480569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4480878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4481110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4481348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4481747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4482126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4482231Z ok (5.417s)
2023-01-11T22:55:24.4482252Z 
2023-01-11T22:55:24.4482514Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4482626Z Ran 1 test in 5.417s
2023-01-11T22:55:24.4482646Z 
2023-01-11T22:55:24.4482737Z OK
2023-01-11T22:55:24.4482757Z 
2023-01-11T22:55:24.4482879Z Generating XML reports...
2023-01-11T22:55:24.4483310Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225354.xml
2023-01-11T22:55:24.4483732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4483907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4484265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4484461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4484482Z 
2023-01-11T22:55:24.4484588Z Running tests...
2023-01-11T22:55:24.4484850Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4485159Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4485397Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4485615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95728
2023-01-11T22:55:24.4485821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95729
2023-01-11T22:55:24.4506479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4506725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4507185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4507373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4507754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4507918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4508312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4508501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4508740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4508995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4509234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4509659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4510089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4510502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4510589Z ok (6.721s)
2023-01-11T22:55:24.4510611Z 
2023-01-11T22:55:24.4510946Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4511069Z Ran 1 test in 6.722s
2023-01-11T22:55:24.4511090Z 
2023-01-11T22:55:24.4511183Z OK
2023-01-11T22:55:24.4511203Z 
2023-01-11T22:55:24.4511330Z Generating XML reports...
2023-01-11T22:55:24.4511792Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225402.xml
2023-01-11T22:55:24.4512186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4512367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4512752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4512950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4512970Z 
2023-01-11T22:55:24.4513080Z Running tests...
2023-01-11T22:55:24.4513361Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4513692Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4513970Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4514196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95845
2023-01-11T22:55:24.4514426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95846
2023-01-11T22:55:24.4514816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4514982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4515379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4515581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4515967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4516146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4516543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4516745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4516983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4517219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4517458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4517709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4518135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4518548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4518650Z ok (5.409s)
2023-01-11T22:55:24.4518671Z 
2023-01-11T22:55:24.4518945Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4519123Z Ran 1 test in 5.409s
2023-01-11T22:55:24.4519143Z 
2023-01-11T22:55:24.4519235Z OK
2023-01-11T22:55:24.4519254Z 
2023-01-11T22:55:24.4519365Z Generating XML reports...
2023-01-11T22:55:24.4520058Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225411.xml
2023-01-11T22:55:24.4520427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4520692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4521087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4521276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4521296Z 
2023-01-11T22:55:24.4521405Z Running tests...
2023-01-11T22:55:24.4521672Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4521971Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4522228Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4522446Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95954
2023-01-11T22:55:24.4522660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95955
2023-01-11T22:55:24.4523027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4523199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4523578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4523767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4524127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4524284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4524656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4524843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4525067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4525311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4525532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4525761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4526161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4526556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4526643Z ok (6.623s)
2023-01-11T22:55:24.4526663Z 
2023-01-11T22:55:24.4526924Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4527036Z Ran 1 test in 6.623s
2023-01-11T22:55:24.4527056Z 
2023-01-11T22:55:24.4527148Z OK
2023-01-11T22:55:24.4527168Z 
2023-01-11T22:55:24.4527291Z Generating XML reports...
2023-01-11T22:55:24.4527720Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225419.xml
2023-01-11T22:55:24.4528085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4528257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4528611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4528874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4528894Z 
2023-01-11T22:55:24.4529002Z Running tests...
2023-01-11T22:55:24.4529265Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4529574Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4529865Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4530090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96071
2023-01-11T22:55:24.4530302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96072
2023-01-11T22:55:24.4530656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4530831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4531211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4531400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4531761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4531933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4532304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4532486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4532714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4532941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4533162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4533401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4533795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4534183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4534288Z ok (6.625s)
2023-01-11T22:55:24.4534308Z 
2023-01-11T22:55:24.4534568Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4534679Z Ran 1 test in 6.625s
2023-01-11T22:55:24.4534699Z 
2023-01-11T22:55:24.4534790Z OK
2023-01-11T22:55:24.4534810Z 
2023-01-11T22:55:24.4534917Z Generating XML reports...
2023-01-11T22:55:24.4535344Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225428.xml
2023-01-11T22:55:24.4535714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4535887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4536260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4536443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4536463Z 
2023-01-11T22:55:24.4536569Z Running tests...
2023-01-11T22:55:24.4536829Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4537122Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4537366Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4537582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96188
2023-01-11T22:55:24.4537892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96189
2023-01-11T22:55:24.4538261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4538433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4538806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4539037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4539408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4539561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4539927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4540113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4540339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4540578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4540803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4541045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4541443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4541817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4541919Z ok (5.315s)
2023-01-11T22:55:24.4541939Z 
2023-01-11T22:55:24.4542201Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4542313Z Ran 1 test in 5.315s
2023-01-11T22:55:24.4542333Z 
2023-01-11T22:55:24.4542423Z OK
2023-01-11T22:55:24.4542443Z 
2023-01-11T22:55:24.4542566Z Generating XML reports...
2023-01-11T22:55:24.4542992Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225436.xml
2023-01-11T22:55:24.4543357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4543534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4543892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4544080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4544100Z 
2023-01-11T22:55:24.4544208Z Running tests...
2023-01-11T22:55:24.4544468Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4544778Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4545009Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4545223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96297
2023-01-11T22:55:24.4545436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96298
2023-01-11T22:55:24.4545812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4545964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4546338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4546524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4546944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4547115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4547481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4547667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4547894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4548166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4548392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4548626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4549024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4549417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4549518Z ok (6.619s)
2023-01-11T22:55:24.4549538Z 
2023-01-11T22:55:24.4549796Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4549908Z Ran 1 test in 6.619s
2023-01-11T22:55:24.4549928Z 
2023-01-11T22:55:24.4550002Z OK
2023-01-11T22:55:24.4550038Z 
2023-01-11T22:55:24.4550147Z Generating XML reports...
2023-01-11T22:55:24.4550575Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225444.xml
2023-01-11T22:55:24.4550938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4551111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4551489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4551675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4551695Z 
2023-01-11T22:55:24.4551803Z Running tests...
2023-01-11T22:55:24.4552059Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4552351Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4552593Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4552807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96418
2023-01-11T22:55:24.4553018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96419
2023-01-11T22:55:24.4553383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4553560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4553932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4554116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4554457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4554626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4555000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4555184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4555410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4555635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4555956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4556194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4556616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4556990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4557122Z ok (11.349s)
2023-01-11T22:55:24.4557145Z 
2023-01-11T22:55:24.4557422Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4557529Z Ran 1 test in 11.349s
2023-01-11T22:55:24.4557573Z 
2023-01-11T22:55:24.4557641Z OK
2023-01-11T22:55:24.4557667Z 
2023-01-11T22:55:24.4557785Z Generating XML reports...
2023-01-11T22:55:24.4558213Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225453.xml
2023-01-11T22:55:24.4558581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4558753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4559111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4559296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4559320Z 
2023-01-11T22:55:24.4559664Z Running tests...
2023-01-11T22:55:24.4559941Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4560258Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4560494Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4560716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96539
2023-01-11T22:55:24.4560928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96540
2023-01-11T22:55:24.4561275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4561448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4561821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4562012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4562372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4562542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4562908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4563094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4563320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:55:24.4563545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:55:24.4563766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:55:24.4564009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4564406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4564794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:55:24.4564895Z ok (5.539s)
2023-01-11T22:55:24.4564914Z 
2023-01-11T22:55:24.4565262Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4565375Z Ran 1 test in 5.539s
2023-01-11T22:55:24.4565396Z 
2023-01-11T22:55:24.4565470Z OK
2023-01-11T22:55:24.4565508Z 
2023-01-11T22:55:24.4565615Z Generating XML reports...
2023-01-11T22:55:24.4566042Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225507.xml
2023-01-11T22:55:24.4566404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4566638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4567025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4567211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4567232Z 
2023-01-11T22:55:24.4567340Z Running tests...
2023-01-11T22:55:24.4567606Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4567895Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4568120Z   test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4568362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4568754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4568993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4569380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4569619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4570006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4570235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4570611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4570713Z ok (1.616s)
2023-01-11T22:55:24.4570732Z 
2023-01-11T22:55:24.4570992Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4571106Z Ran 1 test in 1.617s
2023-01-11T22:55:24.4571126Z 
2023-01-11T22:55:24.4571219Z OK
2023-01-11T22:55:24.4571237Z 
2023-01-11T22:55:24.4571359Z Generating XML reports...
2023-01-11T22:55:24.4571771Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20230111225515.xml
2023-01-11T22:55:24.4572136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T22:55:24.4572297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:55:24.4572674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:55:24.4572862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:55:24.4572882Z 
2023-01-11T22:55:24.4572988Z Running tests...
2023-01-11T22:55:24.4573246Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4573555Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:55:24.4573789Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:55:24.4574028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4574420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4574711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:55:24.4575105Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:55:24.4575206Z ok (4.648s)
2023-01-11T22:55:24.4575226Z 
2023-01-11T22:55:24.4575484Z ----------------------------------------------------------------------
2023-01-11T22:55:24.4575593Z Ran 1 test in 4.648s
2023-01-11T22:55:24.4575657Z 
2023-01-11T22:55:24.4575753Z OK
2023-01-11T22:55:24.4575772Z 
2023-01-11T22:55:24.4575896Z Generating XML reports...
2023-01-11T22:55:24.4576288Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20230111225518.xml
2023-01-11T22:55:24.4576308Z 
2023-01-11T22:55:24.4576765Z ##[endgroup]
2023-01-11T22:55:24.4577195Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_fkrwsvpj)
2023-01-11T22:55:24.4577220Z 
2023-01-11T22:55:24.6332705Z 
2023-01-11T22:55:24.6333014Z real	102m14.218s
2023-01-11T22:55:24.6333266Z user	168m36.390s
2023-01-11T22:55:24.6333445Z sys	83m7.213s
2023-01-11T22:55:24.6333600Z + assert_git_not_dirty
2023-01-11T22:55:24.6333986Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2023-01-11T22:55:24.6334353Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2023-01-11T22:55:24.6336663Z ++ git status --porcelain
2023-01-11T22:55:25.4106449Z + git_status=
2023-01-11T22:55:25.4106858Z + [[ -n '' ]]
2023-01-11T22:55:25.4107262Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2023-01-11T22:55:25.4107532Z + [[ 2 == 1 ]]
2023-01-11T22:55:25.4107755Z + [[ 2 == 1 ]]
2023-01-11T22:55:25.4171170Z ##[group]Run cat test/**/*.log || true
2023-01-11T22:55:25.4171501Z cat test/**/*.log || true
2023-01-11T22:55:25.4184515Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:55:25.4184818Z env:
2023-01-11T22:55:25.4185061Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:25.4185311Z   GPU_FLAG: --gpus all
2023-01-11T22:55:25.4185670Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:25.4186018Z ##[endgroup]
2023-01-11T22:55:25.4231635Z cat: test/**/*.log: No such file or directory
2023-01-11T22:55:25.4263516Z Prepare all required actions
2023-01-11T22:55:25.4263953Z Getting action download info
2023-01-11T22:55:25.5966664Z ##[group]Run ./.github/actions/get-workflow-job-id
2023-01-11T22:55:25.5966939Z with:
2023-01-11T22:55:25.5967415Z   github-token: ***
2023-01-11T22:55:25.5967664Z env:
2023-01-11T22:55:25.5967889Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:25.5968159Z   GPU_FLAG: --gpus all
2023-01-11T22:55:25.5968531Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:25.5968866Z ##[endgroup]
2023-01-11T22:55:25.6005169Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T22:55:25.6005497Z with:
2023-01-11T22:55:25.6005725Z   shell: bash
2023-01-11T22:55:25.6005951Z   timeout_minutes: 10
2023-01-11T22:55:25.6006203Z   max_attempts: 5
2023-01-11T22:55:25.6006456Z   retry_wait_seconds: 30
2023-01-11T22:55:25.6006990Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2023-01-11T22:55:25.6007515Z   polling_interval_seconds: 1
2023-01-11T22:55:25.6007771Z   warning_on_retry: true
2023-01-11T22:55:25.6008039Z   continue_on_error: false
2023-01-11T22:55:25.6008288Z env:
2023-01-11T22:55:25.6008562Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:25.6008835Z   GPU_FLAG: --gpus all
2023-01-11T22:55:25.6009206Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:25.6009719Z   GITHUB_TOKEN: ***
2023-01-11T22:55:25.6010090Z ##[endgroup]
2023-01-11T22:55:25.6690025Z + python3 -m pip install requests==2.26.0
2023-01-11T22:55:25.9844392Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:55:26.1234709Z Collecting requests==2.26.0
2023-01-11T22:55:26.1424364Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2023-01-11T22:55:26.2269600Z Collecting certifi>=2017.4.17
2023-01-11T22:55:26.2315200Z   Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
2023-01-11T22:55:26.4112532Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2023-01-11T22:55:26.4153725Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2023-01-11T22:55:26.5242501Z Collecting urllib3<1.27,>=1.21.1
2023-01-11T22:55:26.5286738Z   Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
2023-01-11T22:55:26.5969969Z Collecting idna<4,>=2.5; python_version >= "3"
2023-01-11T22:55:26.6011452Z   Downloading idna-3.4-py3-none-any.whl (61 kB)
2023-01-11T22:55:26.6892393Z Installing collected packages: certifi, charset-normalizer, urllib3, idna, requests
2023-01-11T22:55:26.7341296Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2023-01-11T22:55:26.7341930Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T22:55:26.9521902Z Successfully installed certifi-2022.12.7 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.14
2023-01-11T22:55:27.0014045Z ++ python3 .github/scripts/get_workflow_job_id.py 3896099317 i-03a86a33898d1fa3b
2023-01-11T22:55:28.5581412Z + GHA_WORKFLOW_JOB_ID=10589291501
2023-01-11T22:55:28.5582270Z + echo job-id=10589291501
2023-01-11T22:55:28.6668164Z Command completed after 1 attempt(s).
2023-01-11T22:55:28.6836896Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2023-01-11T22:55:28.6837302Z kill "$MONITOR_SCRIPT_PID"
2023-01-11T22:55:28.6851999Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:55:28.6852336Z env:
2023-01-11T22:55:28.6852596Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:28.6852854Z   GPU_FLAG: --gpus all
2023-01-11T22:55:28.6853234Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:28.6853616Z   MONITOR_SCRIPT_PID: 60115
2023-01-11T22:55:28.6853868Z ##[endgroup]
2023-01-11T22:55:28.6968380Z Prepare all required actions
2023-01-11T22:55:28.6968791Z Getting action download info
2023-01-11T22:55:29.3635539Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce)
2023-01-11T22:55:29.5141635Z ##[group]Run ./.github/actions/upload-test-artifacts
2023-01-11T22:55:29.5141918Z with:
2023-01-11T22:55:29.5142270Z   file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501
2023-01-11T22:55:29.5142613Z env:
2023-01-11T22:55:29.5142835Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:29.5143099Z   GPU_FLAG: --gpus all
2023-01-11T22:55:29.5143459Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:29.5143796Z ##[endgroup]
2023-01-11T22:55:29.5172257Z ##[group]Run # Remove any previous test jsons if they exist
2023-01-11T22:55:29.5172613Z # Remove any previous test jsons if they exist
2023-01-11T22:55:29.5172927Z rm -f test-jsons-*.zip
2023-01-11T22:55:29.5173300Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2023-01-11T22:55:29.5185284Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:55:29.5185579Z env:
2023-01-11T22:55:29.5185822Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:29.5186071Z   GPU_FLAG: --gpus all
2023-01-11T22:55:29.5186430Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:29.5186895Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501
2023-01-11T22:55:29.5187231Z ##[endgroup]
2023-01-11T22:55:29.5348911Z   adding: test/allowlist_for_publicAPI.json (deflated 78%)
2023-01-11T22:55:29.5383766Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2023-01-11T22:55:29.5390988Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2023-01-11T22:55:29.5392204Z   adding: test/.pytorch-slow-tests.json (deflated 74%)
2023-01-11T22:55:29.5398212Z   adding: test/.pytorch-disabled-tests.json (deflated 84%)
2023-01-11T22:55:29.5443795Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T22:55:29.5444167Z # Remove any previous test reports if they exist
2023-01-11T22:55:29.5444490Z rm -f test-reports-*.zip
2023-01-11T22:55:29.5444852Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2023-01-11T22:55:29.5456861Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:55:29.5457157Z env:
2023-01-11T22:55:29.5457399Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:29.5457652Z   GPU_FLAG: --gpus all
2023-01-11T22:55:29.5458013Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:29.5458483Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501
2023-01-11T22:55:29.5458821Z ##[endgroup]
2023-01-11T22:55:29.5615646Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211321.xml (deflated 43%)
2023-01-11T22:55:29.5616568Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211323.xml (deflated 44%)
2023-01-11T22:55:29.5617447Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211325.xml (deflated 42%)
2023-01-11T22:55:29.5618325Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211334.xml (deflated 42%)
2023-01-11T22:55:29.5619185Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211343.xml (deflated 42%)
2023-01-11T22:55:29.5620068Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211351.xml (deflated 42%)
2023-01-11T22:55:29.5621066Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211402.xml (deflated 43%)
2023-01-11T22:55:29.5621967Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211405.xml (deflated 44%)
2023-01-11T22:55:29.5622832Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211407.xml (deflated 42%)
2023-01-11T22:55:29.5623672Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211415.xml (deflated 42%)
2023-01-11T22:55:29.5624528Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211424.xml (deflated 42%)
2023-01-11T22:55:29.5625391Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111211433.xml (deflated 42%)
2023-01-11T22:55:29.5626214Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211445.xml (deflated 41%)
2023-01-11T22:55:29.5626986Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211454.xml (deflated 42%)
2023-01-11T22:55:29.5627760Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211457.xml (deflated 42%)
2023-01-11T22:55:29.5628537Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211503.xml (deflated 43%)
2023-01-11T22:55:29.5629309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211508.xml (deflated 42%)
2023-01-11T22:55:29.5630178Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211510.xml (deflated 42%)
2023-01-11T22:55:29.5630939Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211512.xml (deflated 40%)
2023-01-11T22:55:29.5631718Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211523.xml (deflated 40%)
2023-01-11T22:55:29.5632479Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211532.xml (deflated 40%)
2023-01-11T22:55:29.5633250Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211542.xml (deflated 39%)
2023-01-11T22:55:29.5634004Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211551.xml (deflated 39%)
2023-01-11T22:55:29.5634781Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211602.xml (deflated 40%)
2023-01-11T22:55:29.5635551Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211612.xml (deflated 40%)
2023-01-11T22:55:29.5636319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211621.xml (deflated 42%)
2023-01-11T22:55:29.5637069Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211625.xml (deflated 41%)
2023-01-11T22:55:29.5637845Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211632.xml (deflated 42%)
2023-01-11T22:55:29.5638614Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211636.xml (deflated 42%)
2023-01-11T22:55:29.5639944Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211646.xml (deflated 42%)
2023-01-11T22:55:29.5640898Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211648.xml (deflated 46%)
2023-01-11T22:55:29.5641682Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211658.xml (deflated 47%)
2023-01-11T22:55:29.5642795Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211708.xml (deflated 48%)
2023-01-11T22:55:29.5643576Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211718.xml (deflated 46%)
2023-01-11T22:55:29.5644355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211728.xml (deflated 41%)
2023-01-11T22:55:29.5645142Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211730.xml (deflated 44%)
2023-01-11T22:55:29.5645911Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211732.xml (deflated 44%)
2023-01-11T22:55:29.5646679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211735.xml (deflated 43%)
2023-01-11T22:55:29.5647456Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211737.xml (deflated 44%)
2023-01-11T22:55:29.5648228Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211740.xml (deflated 43%)
2023-01-11T22:55:29.5649003Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211742.xml (deflated 41%)
2023-01-11T22:55:29.5649867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211744.xml (deflated 40%)
2023-01-11T22:55:29.5650640Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211755.xml (deflated 40%)
2023-01-11T22:55:29.5651409Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211805.xml (deflated 41%)
2023-01-11T22:55:29.5652174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211807.xml (deflated 41%)
2023-01-11T22:55:29.5652927Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211810.xml (deflated 40%)
2023-01-11T22:55:29.5653692Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211820.xml (deflated 40%)
2023-01-11T22:55:29.5655325Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211830.xml (deflated 42%)
2023-01-11T22:55:29.5657052Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211834.xml (deflated 42%)
2023-01-11T22:55:29.5658761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211839.xml (deflated 40%)
2023-01-11T22:55:29.5660463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211848.xml (deflated 40%)
2023-01-11T22:55:29.5662108Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211859.xml (deflated 40%)
2023-01-11T22:55:29.5663747Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211908.xml (deflated 41%)
2023-01-11T22:55:29.5665418Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211911.xml (deflated 41%)
2023-01-11T22:55:29.5667098Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211913.xml (deflated 41%)
2023-01-11T22:55:29.5668925Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211916.xml (deflated 41%)
2023-01-11T22:55:29.5670588Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211918.xml (deflated 41%)
2023-01-11T22:55:29.5672245Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211920.xml (deflated 41%)
2023-01-11T22:55:29.5673914Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211923.xml (deflated 41%)
2023-01-11T22:55:29.5675567Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211925.xml (deflated 41%)
2023-01-11T22:55:29.5677226Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211928.xml (deflated 41%)
2023-01-11T22:55:29.5678844Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211930.xml (deflated 42%)
2023-01-11T22:55:29.5681064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211932.xml (deflated 41%)
2023-01-11T22:55:29.5682735Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211935.xml (deflated 41%)
2023-01-11T22:55:29.5684370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211937.xml (deflated 41%)
2023-01-11T22:55:29.5686057Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211940.xml (deflated 42%)
2023-01-11T22:55:29.5687825Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211942.xml (deflated 41%)
2023-01-11T22:55:29.5689459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211944.xml (deflated 41%)
2023-01-11T22:55:29.5691072Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211947.xml (deflated 41%)
2023-01-11T22:55:29.5692669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211949.xml (deflated 41%)
2023-01-11T22:55:29.5694403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211952.xml (deflated 41%)
2023-01-11T22:55:29.5696053Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211954.xml (deflated 41%)
2023-01-11T22:55:29.5697660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211956.xml (deflated 41%)
2023-01-11T22:55:29.5699275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111211959.xml (deflated 42%)
2023-01-11T22:55:29.5700874Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212001.xml (deflated 42%)
2023-01-11T22:55:29.5702493Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212004.xml (deflated 42%)
2023-01-11T22:55:29.5704103Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212006.xml (deflated 43%)
2023-01-11T22:55:29.5705715Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212008.xml (deflated 43%)
2023-01-11T22:55:29.5707391Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212011.xml (deflated 42%)
2023-01-11T22:55:29.5709015Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212013.xml (deflated 41%)
2023-01-11T22:55:29.5710824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212023.xml (deflated 42%)
2023-01-11T22:55:29.5712442Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212025.xml (deflated 41%)
2023-01-11T22:55:29.5714056Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212027.xml (deflated 41%)
2023-01-11T22:55:29.5715718Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212030.xml (deflated 41%)
2023-01-11T22:55:29.5753673Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212039.xml (deflated 40%)
2023-01-11T22:55:29.5755428Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212047.xml (deflated 41%)
2023-01-11T22:55:29.5757195Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212056.xml (deflated 42%)
2023-01-11T22:55:29.5758865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212059.xml (deflated 42%)
2023-01-11T22:55:29.5760936Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212101.xml (deflated 41%)
2023-01-11T22:55:29.5762593Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212110.xml (deflated 42%)
2023-01-11T22:55:29.5764271Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212120.xml (deflated 43%)
2023-01-11T22:55:29.5766080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212122.xml (deflated 42%)
2023-01-11T22:55:29.5767757Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212132.xml (deflated 43%)
2023-01-11T22:55:29.5769421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212134.xml (deflated 42%)
2023-01-11T22:55:29.5771079Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212141.xml (deflated 43%)
2023-01-11T22:55:29.5772692Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212144.xml (deflated 42%)
2023-01-11T22:55:29.5774346Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212146.xml (deflated 42%)
2023-01-11T22:55:29.5776012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212150.xml (deflated 41%)
2023-01-11T22:55:29.5777670Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212200.xml (deflated 43%)
2023-01-11T22:55:29.5779298Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212203.xml (deflated 42%)
2023-01-11T22:55:29.5780958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212207.xml (deflated 42%)
2023-01-11T22:55:29.5782591Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212209.xml (deflated 42%)
2023-01-11T22:55:29.5784220Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212216.xml (deflated 42%)
2023-01-11T22:55:29.5785861Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212218.xml (deflated 42%)
2023-01-11T22:55:29.5787667Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212221.xml (deflated 41%)
2023-01-11T22:55:29.5789292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212230.xml (deflated 41%)
2023-01-11T22:55:29.5790928Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212240.xml (deflated 42%)
2023-01-11T22:55:29.5792544Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212242.xml (deflated 41%)
2023-01-11T22:55:29.5794184Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212252.xml (deflated 42%)
2023-01-11T22:55:29.5795908Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212254.xml (deflated 42%)
2023-01-11T22:55:29.5797546Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212301.xml (deflated 41%)
2023-01-11T22:55:29.5799184Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212310.xml (deflated 42%)
2023-01-11T22:55:29.5801246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212317.xml (deflated 42%)
2023-01-11T22:55:29.5802926Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212320.xml (deflated 43%)
2023-01-11T22:55:29.5804554Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212322.xml (deflated 41%)
2023-01-11T22:55:29.5806331Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212332.xml (deflated 42%)
2023-01-11T22:55:29.5807995Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212335.xml (deflated 42%)
2023-01-11T22:55:29.5809660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212341.xml (deflated 42%)
2023-01-11T22:55:29.5811316Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212344.xml (deflated 42%)
2023-01-11T22:55:29.5812945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212351.xml (deflated 43%)
2023-01-11T22:55:29.5814526Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212353.xml (deflated 43%)
2023-01-11T22:55:29.5816181Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212355.xml (deflated 43%)
2023-01-11T22:55:29.5817839Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212358.xml (deflated 41%)
2023-01-11T22:55:29.5819507Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212400.xml (deflated 41%)
2023-01-11T22:55:29.5821123Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212402.xml (deflated 41%)
2023-01-11T22:55:29.5822772Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212409.xml (deflated 41%)
2023-01-11T22:55:29.5824421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212418.xml (deflated 41%)
2023-01-11T22:55:29.5826073Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212425.xml (deflated 41%)
2023-01-11T22:55:29.5827709Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212433.xml (deflated 41%)
2023-01-11T22:55:29.5829473Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212440.xml (deflated 40%)
2023-01-11T22:55:29.5831093Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212448.xml (deflated 41%)
2023-01-11T22:55:29.5832754Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212458.xml (deflated 41%)
2023-01-11T22:55:29.5834343Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212505.xml (deflated 42%)
2023-01-11T22:55:29.5835987Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212507.xml (deflated 42%)
2023-01-11T22:55:29.5837631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212511.xml (deflated 42%)
2023-01-11T22:55:29.5839261Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212514.xml (deflated 42%)
2023-01-11T22:55:29.5841277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212516.xml (deflated 43%)
2023-01-11T22:55:29.5842952Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212518.xml (deflated 42%)
2023-01-11T22:55:29.5844618Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212522.xml (deflated 42%)
2023-01-11T22:55:29.5846260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212527.xml (deflated 42%)
2023-01-11T22:55:29.5848066Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212531.xml (deflated 41%)
2023-01-11T22:55:29.5849706Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212541.xml (deflated 40%)
2023-01-11T22:55:29.5851313Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212550.xml (deflated 40%)
2023-01-11T22:55:29.5852950Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212601.xml (deflated 40%)
2023-01-11T22:55:29.5854564Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212609.xml (deflated 40%)
2023-01-11T22:55:29.5856241Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212618.xml (deflated 42%)
2023-01-11T22:55:29.5857915Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212622.xml (deflated 42%)
2023-01-11T22:55:29.5859570Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212627.xml (deflated 40%)
2023-01-11T22:55:29.5861154Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212636.xml (deflated 40%)
2023-01-11T22:55:29.5862762Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212645.xml (deflated 40%)
2023-01-11T22:55:29.5864392Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212655.xml (deflated 40%)
2023-01-11T22:55:29.5865978Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212704.xml (deflated 42%)
2023-01-11T22:55:29.5867557Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212708.xml (deflated 42%)
2023-01-11T22:55:29.5869238Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212711.xml (deflated 42%)
2023-01-11T22:55:29.5871037Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212715.xml (deflated 41%)
2023-01-11T22:55:29.5872691Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212725.xml (deflated 42%)
2023-01-11T22:55:29.5874259Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212729.xml (deflated 42%)
2023-01-11T22:55:29.5875888Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212733.xml (deflated 40%)
2023-01-11T22:55:29.5877510Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212743.xml (deflated 40%)
2023-01-11T22:55:29.5879143Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212754.xml (deflated 42%)
2023-01-11T22:55:29.5881157Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212758.xml (deflated 41%)
2023-01-11T22:55:29.5882729Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212808.xml (deflated 42%)
2023-01-11T22:55:29.5884358Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212810.xml (deflated 42%)
2023-01-11T22:55:29.5885919Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212813.xml (deflated 42%)
2023-01-11T22:55:29.5887512Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212815.xml (deflated 41%)
2023-01-11T22:55:29.5889242Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212818.xml (deflated 41%)
2023-01-11T22:55:29.5890867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212820.xml (deflated 41%)
2023-01-11T22:55:29.5892455Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212822.xml (deflated 41%)
2023-01-11T22:55:29.5894048Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212825.xml (deflated 41%)
2023-01-11T22:55:29.5895713Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212827.xml (deflated 41%)
2023-01-11T22:55:29.5897340Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212830.xml (deflated 41%)
2023-01-11T22:55:29.5898940Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212832.xml (deflated 42%)
2023-01-11T22:55:29.5900536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212834.xml (deflated 42%)
2023-01-11T22:55:29.5902127Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212837.xml (deflated 42%)
2023-01-11T22:55:29.5903759Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212841.xml (deflated 41%)
2023-01-11T22:55:29.5905477Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212850.xml (deflated 40%)
2023-01-11T22:55:29.5907123Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212900.xml (deflated 41%)
2023-01-11T22:55:29.5908751Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212903.xml (deflated 40%)
2023-01-11T22:55:29.5910552Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212913.xml (deflated 40%)
2023-01-11T22:55:29.5912195Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212921.xml (deflated 40%)
2023-01-11T22:55:29.5913847Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212940.xml (deflated 41%)
2023-01-11T22:55:29.5915482Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212950.xml (deflated 41%)
2023-01-11T22:55:29.5917128Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213001.xml (deflated 42%)
2023-01-11T22:55:29.5918790Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213011.xml (deflated 41%)
2023-01-11T22:55:29.5920841Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213021.xml (deflated 42%)
2023-01-11T22:55:29.5922440Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213025.xml (deflated 42%)
2023-01-11T22:55:29.5924056Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213029.xml (deflated 42%)
2023-01-11T22:55:29.5925682Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213039.xml (deflated 40%)
2023-01-11T22:55:29.5927334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213048.xml (deflated 42%)
2023-01-11T22:55:29.5929149Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213052.xml (deflated 40%)
2023-01-11T22:55:29.5930792Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213101.xml (deflated 42%)
2023-01-11T22:55:29.5932417Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213105.xml (deflated 41%)
2023-01-11T22:55:29.5934062Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213115.xml (deflated 41%)
2023-01-11T22:55:29.5935702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213125.xml (deflated 41%)
2023-01-11T22:55:29.5937318Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213134.xml (deflated 42%)
2023-01-11T22:55:29.5938944Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213138.xml (deflated 42%)
2023-01-11T22:55:29.5940548Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213142.xml (deflated 42%)
2023-01-11T22:55:29.5942209Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213146.xml (deflated 41%)
2023-01-11T22:55:29.5943793Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213156.xml (deflated 41%)
2023-01-11T22:55:29.5945402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213206.xml (deflated 40%)
2023-01-11T22:55:29.5947018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213213.xml (deflated 41%)
2023-01-11T22:55:29.5948594Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213220.xml (deflated 42%)
2023-01-11T22:55:29.5950247Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213224.xml (deflated 42%)
2023-01-11T22:55:29.5952013Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213228.xml (deflated 40%)
2023-01-11T22:55:29.5953623Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213235.xml (deflated 41%)
2023-01-11T22:55:29.5955241Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213237.xml (deflated 42%)
2023-01-11T22:55:29.5956889Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213240.xml (deflated 40%)
2023-01-11T22:55:29.5958545Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213250.xml (deflated 42%)
2023-01-11T22:55:29.5960695Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213253.xml (deflated 42%)
2023-01-11T22:55:29.5962312Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213255.xml (deflated 40%)
2023-01-11T22:55:29.5963921Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213305.xml (deflated 42%)
2023-01-11T22:55:29.5965573Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213309.xml (deflated 41%)
2023-01-11T22:55:29.5967135Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213316.xml (deflated 40%)
2023-01-11T22:55:29.5968764Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213325.xml (deflated 41%)
2023-01-11T22:55:29.5970564Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213332.xml (deflated 40%)
2023-01-11T22:55:29.5972224Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213339.xml (deflated 40%)
2023-01-11T22:55:29.5973865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213346.xml (deflated 40%)
2023-01-11T22:55:29.5975537Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213355.xml (deflated 42%)
2023-01-11T22:55:29.5977177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213357.xml (deflated 42%)
2023-01-11T22:55:29.5978847Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213400.xml (deflated 42%)
2023-01-11T22:55:29.5980492Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213402.xml (deflated 42%)
2023-01-11T22:55:29.5982167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213404.xml (deflated 41%)
2023-01-11T22:55:29.5983833Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213429.xml (deflated 41%)
2023-01-11T22:55:29.5985482Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213454.xml (deflated 42%)
2023-01-11T22:55:29.5987135Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml (deflated 41%)
2023-01-11T22:55:29.5988792Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213459.xml (deflated 41%)
2023-01-11T22:55:29.5990488Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213501.xml (deflated 41%)
2023-01-11T22:55:29.5992145Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213504.xml (deflated 41%)
2023-01-11T22:55:29.5993906Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213506.xml (deflated 41%)
2023-01-11T22:55:29.5995625Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213515.xml (deflated 41%)
2023-01-11T22:55:29.5997291Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213523.xml (deflated 41%)
2023-01-11T22:55:29.5998954Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213533.xml (deflated 41%)
2023-01-11T22:55:29.6001036Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213541.xml (deflated 42%)
2023-01-11T22:55:29.6002745Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213545.xml (deflated 42%)
2023-01-11T22:55:29.6004403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213548.xml (deflated 42%)
2023-01-11T22:55:29.6006078Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213550.xml (deflated 41%)
2023-01-11T22:55:29.6007667Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213552.xml (deflated 40%)
2023-01-11T22:55:29.6009333Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213559.xml (deflated 40%)
2023-01-11T22:55:29.6010992Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213606.xml (deflated 41%)
2023-01-11T22:55:29.6012777Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213609.xml (deflated 42%)
2023-01-11T22:55:29.6014411Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213611.xml (deflated 42%)
2023-01-11T22:55:29.6016069Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213615.xml (deflated 40%)
2023-01-11T22:55:29.6017730Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213625.xml (deflated 41%)
2023-01-11T22:55:29.6019408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213635.xml (deflated 41%)
2023-01-11T22:55:29.6021063Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213644.xml (deflated 42%)
2023-01-11T22:55:29.6022693Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213648.xml (deflated 41%)
2023-01-11T22:55:29.6024342Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213652.xml (deflated 40%)
2023-01-11T22:55:29.6026000Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213659.xml (deflated 40%)
2023-01-11T22:55:29.6027666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213706.xml (deflated 42%)
2023-01-11T22:55:29.6029306Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213710.xml (deflated 41%)
2023-01-11T22:55:29.6030959Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213713.xml (deflated 41%)
2023-01-11T22:55:29.6032611Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213715.xml (deflated 41%)
2023-01-11T22:55:29.6034380Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213717.xml (deflated 41%)
2023-01-11T22:55:29.6036176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213720.xml (deflated 41%)
2023-01-11T22:55:29.6037964Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213722.xml (deflated 41%)
2023-01-11T22:55:29.6040089Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213725.xml (deflated 41%)
2023-01-11T22:55:29.6041723Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213727.xml (deflated 41%)
2023-01-11T22:55:29.6043357Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml (deflated 41%)
2023-01-11T22:55:29.6044955Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213732.xml (deflated 42%)
2023-01-11T22:55:29.6046541Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213734.xml (deflated 41%)
2023-01-11T22:55:29.6048148Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213743.xml (deflated 42%)
2023-01-11T22:55:29.6049689Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213746.xml (deflated 40%)
2023-01-11T22:55:29.6051275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213756.xml (deflated 40%)
2023-01-11T22:55:29.6053044Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213805.xml (deflated 41%)
2023-01-11T22:55:29.6054770Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213808.xml (deflated 41%)
2023-01-11T22:55:29.6056537Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213817.xml (deflated 41%)
2023-01-11T22:55:29.6058203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213826.xml (deflated 41%)
2023-01-11T22:55:29.6059866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213828.xml (deflated 42%)
2023-01-11T22:55:29.6061500Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213830.xml (deflated 41%)
2023-01-11T22:55:29.6063103Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213833.xml (deflated 41%)
2023-01-11T22:55:29.6064777Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213835.xml (deflated 40%)
2023-01-11T22:55:29.6066436Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213846.xml (deflated 41%)
2023-01-11T22:55:29.6068085Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213856.xml (deflated 41%)
2023-01-11T22:55:29.6069679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213858.xml (deflated 42%)
2023-01-11T22:55:29.6071276Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213901.xml (deflated 41%)
2023-01-11T22:55:29.6072842Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213903.xml (deflated 43%)
2023-01-11T22:55:29.6074465Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213906.xml (deflated 42%)
2023-01-11T22:55:29.6076205Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213908.xml (deflated 42%)
2023-01-11T22:55:29.6077875Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213910.xml (deflated 42%)
2023-01-11T22:55:29.6080186Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213913.xml (deflated 43%)
2023-01-11T22:55:29.6081184Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213915.xml (deflated 41%)
2023-01-11T22:55:29.6081977Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213924.xml (deflated 40%)
2023-01-11T22:55:29.6082778Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213933.xml (deflated 41%)
2023-01-11T22:55:29.6083567Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213942.xml (deflated 42%)
2023-01-11T22:55:29.6084343Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213944.xml (deflated 43%)
2023-01-11T22:55:29.6085100Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213946.xml (deflated 42%)
2023-01-11T22:55:29.6085879Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213949.xml (deflated 43%)
2023-01-11T22:55:29.6086646Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213951.xml (deflated 42%)
2023-01-11T22:55:29.6087591Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213954.xml (deflated 42%)
2023-01-11T22:55:29.6088367Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213956.xml (deflated 41%)
2023-01-11T22:55:29.6089127Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214006.xml (deflated 41%)
2023-01-11T22:55:29.6089895Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214008.xml (deflated 40%)
2023-01-11T22:55:29.6090660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214018.xml (deflated 41%)
2023-01-11T22:55:29.6091429Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214028.xml (deflated 40%)
2023-01-11T22:55:29.6092176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214042.xml (deflated 40%)
2023-01-11T22:55:29.6092929Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214058.xml (deflated 41%)
2023-01-11T22:55:29.6093686Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214108.xml (deflated 42%)
2023-01-11T22:55:29.6094502Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214110.xml (deflated 42%)
2023-01-11T22:55:29.6095263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214117.xml (deflated 43%)
2023-01-11T22:55:29.6096030Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214121.xml (deflated 42%)
2023-01-11T22:55:29.6096795Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214123.xml (deflated 42%)
2023-01-11T22:55:29.6097553Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214126.xml (deflated 40%)
2023-01-11T22:55:29.6098380Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214136.xml (deflated 40%)
2023-01-11T22:55:29.6099142Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214146.xml (deflated 40%)
2023-01-11T22:55:29.6099888Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214155.xml (deflated 39%)
2023-01-11T22:55:29.6100651Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214205.xml (deflated 39%)
2023-01-11T22:55:29.6101402Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214215.xml (deflated 40%)
2023-01-11T22:55:29.6102171Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214225.xml (deflated 40%)
2023-01-11T22:55:29.6102934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214234.xml (deflated 42%)
2023-01-11T22:55:29.6103700Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214239.xml (deflated 41%)
2023-01-11T22:55:29.6104449Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214245.xml (deflated 42%)
2023-01-11T22:55:29.6105210Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214250.xml (deflated 42%)
2023-01-11T22:55:29.6105974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214259.xml (deflated 42%)
2023-01-11T22:55:29.6106801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214302.xml (deflated 46%)
2023-01-11T22:55:29.6107549Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214311.xml (deflated 47%)
2023-01-11T22:55:29.6108320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214321.xml (deflated 48%)
2023-01-11T22:55:29.6109079Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214331.xml (deflated 46%)
2023-01-11T22:55:29.6109850Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214341.xml (deflated 41%)
2023-01-11T22:55:29.6110599Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214344.xml (deflated 44%)
2023-01-11T22:55:29.6111367Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214346.xml (deflated 43%)
2023-01-11T22:55:29.6112135Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214348.xml (deflated 43%)
2023-01-11T22:55:29.6112903Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214351.xml (deflated 44%)
2023-01-11T22:55:29.6113646Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214353.xml (deflated 43%)
2023-01-11T22:55:29.6114411Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214356.xml (deflated 41%)
2023-01-11T22:55:29.6115170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214358.xml (deflated 40%)
2023-01-11T22:55:29.6115937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214408.xml (deflated 40%)
2023-01-11T22:55:29.6116739Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214419.xml (deflated 41%)
2023-01-11T22:55:29.6117521Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214421.xml (deflated 41%)
2023-01-11T22:55:29.6118285Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214424.xml (deflated 40%)
2023-01-11T22:55:29.6119048Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214434.xml (deflated 40%)
2023-01-11T22:55:29.6120346Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214444.xml (deflated 42%)
2023-01-11T22:55:29.6121133Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214448.xml (deflated 42%)
2023-01-11T22:55:29.6121905Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214452.xml (deflated 40%)
2023-01-11T22:55:29.6122676Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214502.xml (deflated 41%)
2023-01-11T22:55:29.6123414Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214513.xml (deflated 40%)
2023-01-11T22:55:29.6124179Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214522.xml (deflated 41%)
2023-01-11T22:55:29.6124945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214524.xml (deflated 41%)
2023-01-11T22:55:29.6125811Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214527.xml (deflated 41%)
2023-01-11T22:55:29.6126558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214529.xml (deflated 41%)
2023-01-11T22:55:29.6127329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214532.xml (deflated 41%)
2023-01-11T22:55:29.6128091Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214534.xml (deflated 41%)
2023-01-11T22:55:29.6128851Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214536.xml (deflated 41%)
2023-01-11T22:55:29.6129613Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214539.xml (deflated 41%)
2023-01-11T22:55:29.6130358Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214541.xml (deflated 41%)
2023-01-11T22:55:29.6131132Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214544.xml (deflated 41%)
2023-01-11T22:55:29.6131896Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214546.xml (deflated 41%)
2023-01-11T22:55:29.6132661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214548.xml (deflated 41%)
2023-01-11T22:55:29.6133408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214551.xml (deflated 41%)
2023-01-11T22:55:29.6134174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214553.xml (deflated 41%)
2023-01-11T22:55:29.6134934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214556.xml (deflated 41%)
2023-01-11T22:55:29.6135710Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214558.xml (deflated 41%)
2023-01-11T22:55:29.6136533Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214600.xml (deflated 41%)
2023-01-11T22:55:29.6137315Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214603.xml (deflated 41%)
2023-01-11T22:55:29.6138081Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214605.xml (deflated 41%)
2023-01-11T22:55:29.6138844Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214607.xml (deflated 41%)
2023-01-11T22:55:29.6139742Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214610.xml (deflated 41%)
2023-01-11T22:55:29.6140514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214612.xml (deflated 42%)
2023-01-11T22:55:29.6141277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214615.xml (deflated 41%)
2023-01-11T22:55:29.6142041Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214617.xml (deflated 41%)
2023-01-11T22:55:29.6142790Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214620.xml (deflated 43%)
2023-01-11T22:55:29.6143548Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214622.xml (deflated 42%)
2023-01-11T22:55:29.6144311Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214624.xml (deflated 41%)
2023-01-11T22:55:29.6145136Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214627.xml (deflated 41%)
2023-01-11T22:55:29.6145885Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214636.xml (deflated 41%)
2023-01-11T22:55:29.6146655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214639.xml (deflated 41%)
2023-01-11T22:55:29.6147416Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214641.xml (deflated 41%)
2023-01-11T22:55:29.6148172Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214643.xml (deflated 40%)
2023-01-11T22:55:29.6148914Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214652.xml (deflated 41%)
2023-01-11T22:55:29.6149684Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214701.xml (deflated 41%)
2023-01-11T22:55:29.6150449Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214710.xml (deflated 42%)
2023-01-11T22:55:29.6151207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214713.xml (deflated 42%)
2023-01-11T22:55:29.6151953Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214715.xml (deflated 40%)
2023-01-11T22:55:29.6152713Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214724.xml (deflated 41%)
2023-01-11T22:55:29.6153473Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214734.xml (deflated 43%)
2023-01-11T22:55:29.6154240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214736.xml (deflated 41%)
2023-01-11T22:55:29.6154983Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214746.xml (deflated 42%)
2023-01-11T22:55:29.6155807Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214748.xml (deflated 42%)
2023-01-11T22:55:29.6156585Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214755.xml (deflated 42%)
2023-01-11T22:55:29.6157352Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214757.xml (deflated 42%)
2023-01-11T22:55:29.6158118Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214800.xml (deflated 43%)
2023-01-11T22:55:29.6158861Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214804.xml (deflated 41%)
2023-01-11T22:55:29.6160152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214814.xml (deflated 42%)
2023-01-11T22:55:29.6160955Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214817.xml (deflated 42%)
2023-01-11T22:55:29.6161723Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214821.xml (deflated 42%)
2023-01-11T22:55:29.6162475Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214823.xml (deflated 42%)
2023-01-11T22:55:29.6163240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214830.xml (deflated 42%)
2023-01-11T22:55:29.6164002Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214832.xml (deflated 42%)
2023-01-11T22:55:29.6164871Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214835.xml (deflated 41%)
2023-01-11T22:55:29.6165628Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214844.xml (deflated 41%)
2023-01-11T22:55:29.6166396Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214854.xml (deflated 42%)
2023-01-11T22:55:29.6167162Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214856.xml (deflated 40%)
2023-01-11T22:55:29.6167924Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214906.xml (deflated 42%)
2023-01-11T22:55:29.6168672Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214908.xml (deflated 42%)
2023-01-11T22:55:29.6169438Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214915.xml (deflated 41%)
2023-01-11T22:55:29.6170205Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214924.xml (deflated 42%)
2023-01-11T22:55:29.6170970Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214931.xml (deflated 42%)
2023-01-11T22:55:29.6171715Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214933.xml (deflated 42%)
2023-01-11T22:55:29.6172474Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214936.xml (deflated 40%)
2023-01-11T22:55:29.6173233Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214946.xml (deflated 42%)
2023-01-11T22:55:29.6174002Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214949.xml (deflated 42%)
2023-01-11T22:55:29.6174833Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214955.xml (deflated 42%)
2023-01-11T22:55:29.6175610Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214958.xml (deflated 42%)
2023-01-11T22:55:29.6176373Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215004.xml (deflated 42%)
2023-01-11T22:55:29.6177139Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215007.xml (deflated 43%)
2023-01-11T22:55:29.6177890Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215009.xml (deflated 43%)
2023-01-11T22:55:29.6178656Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215012.xml (deflated 41%)
2023-01-11T22:55:29.6179427Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215014.xml (deflated 41%)
2023-01-11T22:55:29.6180192Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215016.xml (deflated 41%)
2023-01-11T22:55:29.6180936Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215023.xml (deflated 41%)
2023-01-11T22:55:29.6181702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215032.xml (deflated 41%)
2023-01-11T22:55:29.6182463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215038.xml (deflated 41%)
2023-01-11T22:55:29.6183285Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215047.xml (deflated 41%)
2023-01-11T22:55:29.6184032Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215054.xml (deflated 40%)
2023-01-11T22:55:29.6184805Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215102.xml (deflated 41%)
2023-01-11T22:55:29.6185568Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215112.xml (deflated 41%)
2023-01-11T22:55:29.6186328Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215119.xml (deflated 41%)
2023-01-11T22:55:29.6187071Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215121.xml (deflated 42%)
2023-01-11T22:55:29.6187836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215125.xml (deflated 42%)
2023-01-11T22:55:29.6188600Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215127.xml (deflated 42%)
2023-01-11T22:55:29.6189364Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215130.xml (deflated 44%)
2023-01-11T22:55:29.6190110Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215132.xml (deflated 42%)
2023-01-11T22:55:29.6190869Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215136.xml (deflated 42%)
2023-01-11T22:55:29.6191636Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215141.xml (deflated 42%)
2023-01-11T22:55:29.6192399Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215145.xml (deflated 40%)
2023-01-11T22:55:29.6193145Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215155.xml (deflated 40%)
2023-01-11T22:55:29.6194010Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215205.xml (deflated 40%)
2023-01-11T22:55:29.6194836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215215.xml (deflated 41%)
2023-01-11T22:55:29.6195600Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215223.xml (deflated 40%)
2023-01-11T22:55:29.6196344Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215232.xml (deflated 42%)
2023-01-11T22:55:29.6197111Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215237.xml (deflated 42%)
2023-01-11T22:55:29.6197883Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215241.xml (deflated 40%)
2023-01-11T22:55:29.6198649Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215250.xml (deflated 40%)
2023-01-11T22:55:29.6199704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215300.xml (deflated 41%)
2023-01-11T22:55:29.6200580Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215310.xml (deflated 40%)
2023-01-11T22:55:29.6201355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215319.xml (deflated 42%)
2023-01-11T22:55:29.6202114Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215323.xml (deflated 42%)
2023-01-11T22:55:29.6202983Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215325.xml (deflated 42%)
2023-01-11T22:55:29.6203738Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215329.xml (deflated 41%)
2023-01-11T22:55:29.6204503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215339.xml (deflated 42%)
2023-01-11T22:55:29.6205259Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215344.xml (deflated 42%)
2023-01-11T22:55:29.6206020Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215348.xml (deflated 40%)
2023-01-11T22:55:29.6206766Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215358.xml (deflated 40%)
2023-01-11T22:55:29.6207531Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215409.xml (deflated 42%)
2023-01-11T22:55:29.6208294Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215413.xml (deflated 40%)
2023-01-11T22:55:29.6209064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215423.xml (deflated 42%)
2023-01-11T22:55:29.6209809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215425.xml (deflated 42%)
2023-01-11T22:55:29.6210566Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215428.xml (deflated 41%)
2023-01-11T22:55:29.6211326Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215430.xml (deflated 41%)
2023-01-11T22:55:29.6212094Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215432.xml (deflated 41%)
2023-01-11T22:55:29.6212836Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215435.xml (deflated 41%)
2023-01-11T22:55:29.6213661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215437.xml (deflated 41%)
2023-01-11T22:55:29.6214439Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215440.xml (deflated 41%)
2023-01-11T22:55:29.6215202Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215442.xml (deflated 41%)
2023-01-11T22:55:29.6215949Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215444.xml (deflated 41%)
2023-01-11T22:55:29.6216705Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215447.xml (deflated 42%)
2023-01-11T22:55:29.6217476Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215449.xml (deflated 42%)
2023-01-11T22:55:29.6218239Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215452.xml (deflated 42%)
2023-01-11T22:55:29.6218984Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215456.xml (deflated 40%)
2023-01-11T22:55:29.6219743Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215505.xml (deflated 40%)
2023-01-11T22:55:29.6220497Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215515.xml (deflated 41%)
2023-01-11T22:55:29.6221256Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215517.xml (deflated 40%)
2023-01-11T22:55:29.6222064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215527.xml (deflated 40%)
2023-01-11T22:55:29.6222830Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215536.xml (deflated 40%)
2023-01-11T22:55:29.6223592Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215555.xml (deflated 41%)
2023-01-11T22:55:29.6224375Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215605.xml (deflated 41%)
2023-01-11T22:55:29.6225119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215615.xml (deflated 41%)
2023-01-11T22:55:29.6225882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215625.xml (deflated 40%)
2023-01-11T22:55:29.6226647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215635.xml (deflated 42%)
2023-01-11T22:55:29.6227411Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215639.xml (deflated 42%)
2023-01-11T22:55:29.6228151Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215644.xml (deflated 42%)
2023-01-11T22:55:29.6228914Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215653.xml (deflated 40%)
2023-01-11T22:55:29.6229671Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215702.xml (deflated 42%)
2023-01-11T22:55:29.6230431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215707.xml (deflated 40%)
2023-01-11T22:55:29.6231179Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215716.xml (deflated 42%)
2023-01-11T22:55:29.6231993Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215720.xml (deflated 41%)
2023-01-11T22:55:29.6232766Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215729.xml (deflated 41%)
2023-01-11T22:55:29.6233527Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215739.xml (deflated 40%)
2023-01-11T22:55:29.6234271Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215748.xml (deflated 42%)
2023-01-11T22:55:29.6235033Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215753.xml (deflated 42%)
2023-01-11T22:55:29.6235798Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215757.xml (deflated 42%)
2023-01-11T22:55:29.6236556Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215801.xml (deflated 41%)
2023-01-11T22:55:29.6237320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215811.xml (deflated 41%)
2023-01-11T22:55:29.6238067Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215820.xml (deflated 40%)
2023-01-11T22:55:29.6238825Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215827.xml (deflated 41%)
2023-01-11T22:55:29.6240022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215834.xml (deflated 42%)
2023-01-11T22:55:29.6240899Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215838.xml (deflated 42%)
2023-01-11T22:55:29.6241646Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215842.xml (deflated 40%)
2023-01-11T22:55:29.6242410Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215849.xml (deflated 41%)
2023-01-11T22:55:29.6243175Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215851.xml (deflated 41%)
2023-01-11T22:55:29.6243937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215854.xml (deflated 40%)
2023-01-11T22:55:29.6244684Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215904.xml (deflated 41%)
2023-01-11T22:55:29.6245440Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215907.xml (deflated 41%)
2023-01-11T22:55:29.6246202Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215909.xml (deflated 40%)
2023-01-11T22:55:29.6246967Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215919.xml (deflated 42%)
2023-01-11T22:55:29.6247714Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215923.xml (deflated 41%)
2023-01-11T22:55:29.6248476Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215930.xml (deflated 40%)
2023-01-11T22:55:29.6249234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215939.xml (deflated 40%)
2023-01-11T22:55:29.6249993Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215946.xml (deflated 40%)
2023-01-11T22:55:29.6250739Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215953.xml (deflated 40%)
2023-01-11T22:55:29.6251572Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220000.xml (deflated 40%)
2023-01-11T22:55:29.6252347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220009.xml (deflated 43%)
2023-01-11T22:55:29.6253111Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220011.xml (deflated 43%)
2023-01-11T22:55:29.6253855Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220014.xml (deflated 42%)
2023-01-11T22:55:29.6254614Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220016.xml (deflated 42%)
2023-01-11T22:55:29.6255378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220018.xml (deflated 41%)
2023-01-11T22:55:29.6256140Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220043.xml (deflated 41%)
2023-01-11T22:55:29.6256884Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220109.xml (deflated 42%)
2023-01-11T22:55:29.6257640Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220111.xml (deflated 42%)
2023-01-11T22:55:29.6258400Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220113.xml (deflated 41%)
2023-01-11T22:55:29.6259163Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220116.xml (deflated 41%)
2023-01-11T22:55:29.6259967Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220118.xml (deflated 41%)
2023-01-11T22:55:29.6260734Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220120.xml (deflated 41%)
2023-01-11T22:55:29.6261497Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220129.xml (deflated 42%)
2023-01-11T22:55:29.6262262Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220137.xml (deflated 41%)
2023-01-11T22:55:29.6263002Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220147.xml (deflated 41%)
2023-01-11T22:55:29.6263763Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220155.xml (deflated 42%)
2023-01-11T22:55:29.6264525Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220159.xml (deflated 42%)
2023-01-11T22:55:29.6265282Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220202.xml (deflated 42%)
2023-01-11T22:55:29.6266031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220204.xml (deflated 42%)
2023-01-11T22:55:29.6266797Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220207.xml (deflated 40%)
2023-01-11T22:55:29.6267558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220213.xml (deflated 41%)
2023-01-11T22:55:29.6268319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220220.xml (deflated 42%)
2023-01-11T22:55:29.6269064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220223.xml (deflated 42%)
2023-01-11T22:55:29.6269826Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220225.xml (deflated 42%)
2023-01-11T22:55:29.6270632Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220229.xml (deflated 41%)
2023-01-11T22:55:29.6271404Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220239.xml (deflated 41%)
2023-01-11T22:55:29.6272146Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220249.xml (deflated 40%)
2023-01-11T22:55:29.6272912Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220258.xml (deflated 42%)
2023-01-11T22:55:29.6273671Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220303.xml (deflated 41%)
2023-01-11T22:55:29.6274433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220307.xml (deflated 40%)
2023-01-11T22:55:29.6275178Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220314.xml (deflated 40%)
2023-01-11T22:55:29.6275942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220320.xml (deflated 42%)
2023-01-11T22:55:29.6276702Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220325.xml (deflated 41%)
2023-01-11T22:55:29.6277461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220327.xml (deflated 42%)
2023-01-11T22:55:29.6278200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220329.xml (deflated 42%)
2023-01-11T22:55:29.6279022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220332.xml (deflated 42%)
2023-01-11T22:55:29.6280243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220334.xml (deflated 41%)
2023-01-11T22:55:29.6281011Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220337.xml (deflated 41%)
2023-01-11T22:55:29.6281757Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220339.xml (deflated 42%)
2023-01-11T22:55:29.6282522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220341.xml (deflated 41%)
2023-01-11T22:55:29.6283288Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220344.xml (deflated 41%)
2023-01-11T22:55:29.6284053Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220346.xml (deflated 42%)
2023-01-11T22:55:29.6284817Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220349.xml (deflated 40%)
2023-01-11T22:55:29.6285560Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220358.xml (deflated 42%)
2023-01-11T22:55:29.6286319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220400.xml (deflated 40%)
2023-01-11T22:55:29.6287061Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220410.xml (deflated 41%)
2023-01-11T22:55:29.6287804Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220420.xml (deflated 41%)
2023-01-11T22:55:29.6288549Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220422.xml (deflated 41%)
2023-01-11T22:55:29.6289366Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220431.xml (deflated 41%)
2023-01-11T22:55:29.6290132Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220440.xml (deflated 42%)
2023-01-11T22:55:29.6290881Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220443.xml (deflated 42%)
2023-01-11T22:55:29.6291619Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220445.xml (deflated 41%)
2023-01-11T22:55:29.6292362Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220448.xml (deflated 41%)
2023-01-11T22:55:29.6293112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220450.xml (deflated 41%)
2023-01-11T22:55:29.6293864Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220500.xml (deflated 40%)
2023-01-11T22:55:29.6294647Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220511.xml (deflated 42%)
2023-01-11T22:55:29.6295396Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220513.xml (deflated 42%)
2023-01-11T22:55:29.6296139Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220516.xml (deflated 41%)
2023-01-11T22:55:29.6296884Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220518.xml (deflated 43%)
2023-01-11T22:55:29.6297699Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220520.xml (deflated 42%)
2023-01-11T22:55:29.6298459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220523.xml (deflated 42%)
2023-01-11T22:55:29.6299220Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220525.xml (deflated 42%)
2023-01-11T22:55:29.6299965Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220528.xml (deflated 43%)
2023-01-11T22:55:29.6300703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220530.xml (deflated 41%)
2023-01-11T22:55:29.6301464Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220538.xml (deflated 40%)
2023-01-11T22:55:29.6302221Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220547.xml (deflated 40%)
2023-01-11T22:55:29.6302985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220556.xml (deflated 43%)
2023-01-11T22:55:29.6303732Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220559.xml (deflated 43%)
2023-01-11T22:55:29.6304492Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220601.xml (deflated 43%)
2023-01-11T22:55:29.6305253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220604.xml (deflated 43%)
2023-01-11T22:55:29.6306007Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220606.xml (deflated 42%)
2023-01-11T22:55:29.6306751Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220608.xml (deflated 42%)
2023-01-11T22:55:29.6307518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220611.xml (deflated 42%)
2023-01-11T22:55:29.6308325Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220621.xml (deflated 41%)
2023-01-11T22:55:29.6309095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220623.xml (deflated 40%)
2023-01-11T22:55:29.6309840Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220633.xml (deflated 41%)
2023-01-11T22:55:29.6310602Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220643.xml (deflated 40%)
2023-01-11T22:55:29.6311364Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220656.xml (deflated 40%)
2023-01-11T22:55:29.6312087Z   adding: test/test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20230111220750.xml (deflated 83%)
2023-01-11T22:55:29.6312790Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20230111220754.xml (deflated 77%)
2023-01-11T22:55:29.6313530Z   adding: test/test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20230111220758.xml (deflated 68%)
2023-01-11T22:55:29.6314293Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20230111220802.xml (deflated 40%)
2023-01-11T22:55:29.6315064Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20230111220806.xml (deflated 71%)
2023-01-11T22:55:29.6315842Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20230111220810.xml (deflated 72%)
2023-01-11T22:55:29.6316794Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20230111220815.xml (deflated 44%)
2023-01-11T22:55:29.6317738Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20230111220821.xml (deflated 43%)
2023-01-11T22:55:29.6318610Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20230111220829.xml (deflated 47%)
2023-01-11T22:55:29.6319846Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_fsdp_model_state/TEST-FsdpModelStateCheckpoint-20230111220837.xml (deflated 60%)
2023-01-11T22:55:29.6320746Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20230111220846.xml (deflated 60%)
2023-01-11T22:55:29.6321564Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20230111220855.xml (deflated 61%)
2023-01-11T22:55:29.6322299Z   adding: test/test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20230111220903.xml (deflated 51%)
2023-01-11T22:55:29.6323090Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20230111220912.xml (deflated 59%)
2023-01-11T22:55:29.6323916Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20230111220924.xml (deflated 54%)
2023-01-11T22:55:29.6324772Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20230111220938.xml (deflated 73%)
2023-01-11T22:55:29.6325604Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20230111220951.xml (deflated 66%)
2023-01-11T22:55:29.6326412Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20230111220951.xml (deflated 61%)
2023-01-11T22:55:29.6327234Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20230111221007.xml (deflated 43%)
2023-01-11T22:55:29.6328173Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20230111221007.xml (deflated 43%)
2023-01-11T22:55:29.6329082Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20230111221023.xml (deflated 65%)
2023-01-11T22:55:29.6329902Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20230111221040.xml (deflated 55%)
2023-01-11T22:55:29.6330659Z   adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20230111221100.xml (deflated 69%)
2023-01-11T22:55:29.6331529Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20230111221119.xml (deflated 67%)
2023-01-11T22:55:29.6332455Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20230111221119.xml (deflated 46%)
2023-01-11T22:55:29.6333474Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111221119.xml (deflated 44%)
2023-01-11T22:55:29.6334427Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20230111221145.xml (deflated 55%)
2023-01-11T22:55:29.6335270Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20230111221145.xml (deflated 81%)
2023-01-11T22:55:29.6336121Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20230111221145.xml (deflated 80%)
2023-01-11T22:55:29.6336998Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20230111221145.xml (deflated 64%)
2023-01-11T22:55:29.6337761Z   adding: test/test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20230111221214.xml (deflated 84%)
2023-01-11T22:55:29.6338510Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20230111221245.xml (deflated 61%)
2023-01-11T22:55:29.6339401Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20230111221319.xml (deflated 79%)
2023-01-11T22:55:29.6340262Z   adding: test/test-reports/python-unittest/distributed._composable.test_compose/TEST-TestFSDPCheckpoint-20230111221356.xml (deflated 75%)
2023-01-11T22:55:29.6341112Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20230111221436.xml (deflated 80%)
2023-01-11T22:55:29.6342039Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20230111221436.xml (deflated 55%)
2023-01-11T22:55:29.6343077Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111221436.xml (deflated 60%)
2023-01-11T22:55:29.6344080Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111221522.xml (deflated 44%)
2023-01-11T22:55:29.6345002Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111221526.xml (deflated 43%)
2023-01-11T22:55:29.6345907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111221530.xml (deflated 44%)
2023-01-11T22:55:29.6346769Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221536.xml (deflated 42%)
2023-01-11T22:55:29.6347664Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221545.xml (deflated 42%)
2023-01-11T22:55:29.6348517Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221554.xml (deflated 42%)
2023-01-11T22:55:29.6349356Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221603.xml (deflated 42%)
2023-01-11T22:55:29.6350170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221613.xml (deflated 42%)
2023-01-11T22:55:29.6351000Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221622.xml (deflated 42%)
2023-01-11T22:55:29.6351841Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221631.xml (deflated 41%)
2023-01-11T22:55:29.6352675Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111221640.xml (deflated 42%)
2023-01-11T22:55:29.6353487Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221653.xml (deflated 42%)
2023-01-11T22:55:29.6354315Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221703.xml (deflated 42%)
2023-01-11T22:55:29.6355146Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221712.xml (deflated 43%)
2023-01-11T22:55:29.6356046Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221721.xml (deflated 43%)
2023-01-11T22:55:29.6356857Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221731.xml (deflated 42%)
2023-01-11T22:55:29.6357688Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221740.xml (deflated 42%)
2023-01-11T22:55:29.6358521Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221749.xml (deflated 42%)
2023-01-11T22:55:29.6359352Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221758.xml (deflated 42%)
2023-01-11T22:55:29.6360623Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111221808.xml (deflated 42%)
2023-01-11T22:55:29.6361373Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221818.xml (deflated 40%)
2023-01-11T22:55:29.6362065Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221822.xml (deflated 39%)
2023-01-11T22:55:29.6362749Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221825.xml (deflated 39%)
2023-01-11T22:55:29.6363418Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111221829.xml (deflated 40%)
2023-01-11T22:55:29.6364098Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111221833.xml (deflated 39%)
2023-01-11T22:55:29.6364774Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111221837.xml (deflated 39%)
2023-01-11T22:55:29.6365482Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111221841.xml (deflated 40%)
2023-01-11T22:55:29.6366190Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111221845.xml (deflated 40%)
2023-01-11T22:55:29.6366976Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20230111221849.xml (deflated 40%)
2023-01-11T22:55:29.6367704Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111221851.xml (deflated 40%)
2023-01-11T22:55:29.6368418Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111221855.xml (deflated 39%)
2023-01-11T22:55:29.6369101Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20230111221859.xml (deflated 39%)
2023-01-11T22:55:29.6369810Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20230111221903.xml (deflated 39%)
2023-01-11T22:55:29.6370518Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111221906.xml (deflated 40%)
2023-01-11T22:55:29.6371226Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111221910.xml (deflated 40%)
2023-01-11T22:55:29.6371915Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221914.xml (deflated 39%)
2023-01-11T22:55:29.6372625Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221918.xml (deflated 39%)
2023-01-11T22:55:29.6373333Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221922.xml (deflated 39%)
2023-01-11T22:55:29.6374033Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111221926.xml (deflated 39%)
2023-01-11T22:55:29.6374702Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111221940.xml (deflated 39%)
2023-01-11T22:55:29.6375468Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111221944.xml (deflated 38%)
2023-01-11T22:55:29.6376143Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221947.xml (deflated 39%)
2023-01-11T22:55:29.6376822Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221951.xml (deflated 39%)
2023-01-11T22:55:29.6377482Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221955.xml (deflated 38%)
2023-01-11T22:55:29.6378151Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111221959.xml (deflated 38%)
2023-01-11T22:55:29.6378808Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222003.xml (deflated 38%)
2023-01-11T22:55:29.6379467Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222007.xml (deflated 39%)
2023-01-11T22:55:29.6380180Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222011.xml (deflated 39%)
2023-01-11T22:55:29.6380846Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111222017.xml (deflated 39%)
2023-01-11T22:55:29.6381568Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20230111222021.xml (deflated 84%)
2023-01-11T22:55:29.6382350Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20230111222136.xml (deflated 91%)
2023-01-11T22:55:29.6383179Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20230111222337.xml (deflated 43%)
2023-01-11T22:55:29.6384051Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20230111222337.xml (deflated 44%)
2023-01-11T22:55:29.6384875Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20230111222337.xml (deflated 60%)
2023-01-11T22:55:29.6385733Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20230111222337.xml (deflated 58%)
2023-01-11T22:55:29.6386538Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20230111222337.xml (deflated 58%)
2023-01-11T22:55:29.6387428Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20230111222337.xml (deflated 61%)
2023-01-11T22:55:29.6388234Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20230111222337.xml (deflated 61%)
2023-01-11T22:55:29.6389049Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20230111222337.xml (deflated 89%)
2023-01-11T22:55:29.6389918Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20230111222337.xml (deflated 69%)
2023-01-11T22:55:29.6390798Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20230111222337.xml (deflated 87%)
2023-01-11T22:55:29.6391706Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20230111222337.xml (deflated 82%)
2023-01-11T22:55:29.6392614Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20230111222337.xml (deflated 62%)
2023-01-11T22:55:29.6393474Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20230111222608.xml (deflated 91%)
2023-01-11T22:55:29.6394425Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20230111222608.xml (deflated 43%)
2023-01-11T22:55:29.6395244Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20230111222946.xml (deflated 95%)
2023-01-11T22:55:29.6395936Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223748.xml (deflated 38%)
2023-01-11T22:55:29.6396601Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223757.xml (deflated 38%)
2023-01-11T22:55:29.6397253Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223805.xml (deflated 39%)
2023-01-11T22:55:29.6397913Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223812.xml (deflated 38%)
2023-01-11T22:55:29.6398545Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223821.xml (deflated 38%)
2023-01-11T22:55:29.6399208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223827.xml (deflated 39%)
2023-01-11T22:55:29.6400304Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223833.xml (deflated 39%)
2023-01-11T22:55:29.6400959Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223839.xml (deflated 39%)
2023-01-11T22:55:29.6401593Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223846.xml (deflated 38%)
2023-01-11T22:55:29.6402239Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223854.xml (deflated 38%)
2023-01-11T22:55:29.6402888Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223903.xml (deflated 37%)
2023-01-11T22:55:29.6403535Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223912.xml (deflated 37%)
2023-01-11T22:55:29.6404172Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223919.xml (deflated 38%)
2023-01-11T22:55:29.6404896Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223928.xml (deflated 38%)
2023-01-11T22:55:29.6405558Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223936.xml (deflated 39%)
2023-01-11T22:55:29.6406208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223942.xml (deflated 38%)
2023-01-11T22:55:29.6406838Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223950.xml (deflated 38%)
2023-01-11T22:55:29.6407484Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111223957.xml (deflated 37%)
2023-01-11T22:55:29.6408133Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224006.xml (deflated 37%)
2023-01-11T22:55:29.6408803Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224014.xml (deflated 38%)
2023-01-11T22:55:29.6409475Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224023.xml (deflated 38%)
2023-01-11T22:55:29.6410160Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224032.xml (deflated 38%)
2023-01-11T22:55:29.6410836Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224041.xml (deflated 38%)
2023-01-11T22:55:29.6411509Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224050.xml (deflated 38%)
2023-01-11T22:55:29.6412165Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224059.xml (deflated 38%)
2023-01-11T22:55:29.6412923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224108.xml (deflated 39%)
2023-01-11T22:55:29.6413596Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224117.xml (deflated 38%)
2023-01-11T22:55:29.6414274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224126.xml (deflated 38%)
2023-01-11T22:55:29.6414931Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224135.xml (deflated 38%)
2023-01-11T22:55:29.6415676Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224144.xml (deflated 41%)
2023-01-11T22:55:29.6416476Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224154.xml (deflated 41%)
2023-01-11T22:55:29.6417261Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224203.xml (deflated 41%)
2023-01-11T22:55:29.6418048Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224213.xml (deflated 41%)
2023-01-11T22:55:29.6418836Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224222.xml (deflated 41%)
2023-01-11T22:55:29.6419623Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224231.xml (deflated 42%)
2023-01-11T22:55:29.6420418Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224240.xml (deflated 41%)
2023-01-11T22:55:29.6421190Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224249.xml (deflated 41%)
2023-01-11T22:55:29.6421968Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224258.xml (deflated 42%)
2023-01-11T22:55:29.6422757Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224307.xml (deflated 44%)
2023-01-11T22:55:29.6423591Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224315.xml (deflated 45%)
2023-01-11T22:55:29.6424372Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224323.xml (deflated 43%)
2023-01-11T22:55:29.6425153Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224332.xml (deflated 43%)
2023-01-11T22:55:29.6425939Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224340.xml (deflated 45%)
2023-01-11T22:55:29.6426720Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224349.xml (deflated 45%)
2023-01-11T22:55:29.6427496Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224357.xml (deflated 46%)
2023-01-11T22:55:29.6428279Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224406.xml (deflated 47%)
2023-01-11T22:55:29.6429060Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224414.xml (deflated 45%)
2023-01-11T22:55:29.6429841Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224422.xml (deflated 46%)
2023-01-11T22:55:29.6430605Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224431.xml (deflated 46%)
2023-01-11T22:55:29.6431377Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224439.xml (deflated 44%)
2023-01-11T22:55:29.6432217Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224448.xml (deflated 44%)
2023-01-11T22:55:29.6432999Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224456.xml (deflated 42%)
2023-01-11T22:55:29.6433762Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224505.xml (deflated 42%)
2023-01-11T22:55:29.6434544Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224514.xml (deflated 42%)
2023-01-11T22:55:29.6435316Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224523.xml (deflated 44%)
2023-01-11T22:55:29.6436094Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224532.xml (deflated 44%)
2023-01-11T22:55:29.6436864Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224541.xml (deflated 42%)
2023-01-11T22:55:29.6437647Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224547.xml (deflated 44%)
2023-01-11T22:55:29.6438420Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224557.xml (deflated 41%)
2023-01-11T22:55:29.6439194Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224606.xml (deflated 41%)
2023-01-11T22:55:29.6440410Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224612.xml (deflated 42%)
2023-01-11T22:55:29.6441201Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224621.xml (deflated 41%)
2023-01-11T22:55:29.6441988Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224630.xml (deflated 42%)
2023-01-11T22:55:29.6442770Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224640.xml (deflated 42%)
2023-01-11T22:55:29.6443609Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224644.xml (deflated 42%)
2023-01-11T22:55:29.6444410Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224647.xml (deflated 42%)
2023-01-11T22:55:29.6445187Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224651.xml (deflated 41%)
2023-01-11T22:55:29.6445973Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224655.xml (deflated 42%)
2023-01-11T22:55:29.6446753Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224659.xml (deflated 42%)
2023-01-11T22:55:29.6447465Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20230111225518.xml (deflated 39%)
2023-01-11T22:55:29.6448199Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224703.xml (deflated 42%)
2023-01-11T22:55:29.6448977Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224713.xml (deflated 41%)
2023-01-11T22:55:29.6449757Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224722.xml (deflated 42%)
2023-01-11T22:55:29.6450521Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224730.xml (deflated 41%)
2023-01-11T22:55:29.6451285Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224740.xml (deflated 41%)
2023-01-11T22:55:29.6452139Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224751.xml (deflated 42%)
2023-01-11T22:55:29.6452923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224757.xml (deflated 41%)
2023-01-11T22:55:29.6453686Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224803.xml (deflated 43%)
2023-01-11T22:55:29.6454458Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224813.xml (deflated 42%)
2023-01-11T22:55:29.6455236Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224822.xml (deflated 42%)
2023-01-11T22:55:29.6456010Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224832.xml (deflated 42%)
2023-01-11T22:55:29.6456779Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224841.xml (deflated 42%)
2023-01-11T22:55:29.6457559Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224847.xml (deflated 42%)
2023-01-11T22:55:29.6458335Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224854.xml (deflated 41%)
2023-01-11T22:55:29.6459114Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224901.xml (deflated 43%)
2023-01-11T22:55:29.6459875Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224908.xml (deflated 42%)
2023-01-11T22:55:29.6460648Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224917.xml (deflated 42%)
2023-01-11T22:55:29.6461432Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224926.xml (deflated 42%)
2023-01-11T22:55:29.6462274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224951.xml (deflated 44%)
2023-01-11T22:55:29.6463049Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225000.xml (deflated 42%)
2023-01-11T22:55:29.6463824Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225009.xml (deflated 42%)
2023-01-11T22:55:29.6464609Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225015.xml (deflated 41%)
2023-01-11T22:55:29.6465389Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225024.xml (deflated 41%)
2023-01-11T22:55:29.6466150Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225033.xml (deflated 42%)
2023-01-11T22:55:29.6466927Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225044.xml (deflated 41%)
2023-01-11T22:55:29.6467691Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225053.xml (deflated 41%)
2023-01-11T22:55:29.6468441Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225100.xml (deflated 41%)
2023-01-11T22:55:29.6469171Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225106.xml (deflated 42%)
2023-01-11T22:55:29.6469918Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225108.xml (deflated 42%)
2023-01-11T22:55:29.6470720Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225115.xml (deflated 41%)
2023-01-11T22:55:29.6471462Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225121.xml (deflated 41%)
2023-01-11T22:55:29.6472192Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225127.xml (deflated 41%)
2023-01-11T22:55:29.6472926Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225133.xml (deflated 42%)
2023-01-11T22:55:29.6473664Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225136.xml (deflated 41%)
2023-01-11T22:55:29.6474533Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225142.xml (deflated 43%)
2023-01-11T22:55:29.6475484Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225149.xml (deflated 42%)
2023-01-11T22:55:29.6476444Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225157.xml (deflated 42%)
2023-01-11T22:55:29.6477402Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225205.xml (deflated 44%)
2023-01-11T22:55:29.6478345Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225212.xml (deflated 42%)
2023-01-11T22:55:29.6479208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20230111225220.xml (deflated 41%)
2023-01-11T22:55:29.6480412Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225222.xml (deflated 39%)
2023-01-11T22:55:29.6481170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225230.xml (deflated 38%)
2023-01-11T22:55:29.6481919Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225239.xml (deflated 38%)
2023-01-11T22:55:29.6482741Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225248.xml (deflated 39%)
2023-01-11T22:55:29.6483479Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225257.xml (deflated 38%)
2023-01-11T22:55:29.6484224Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225306.xml (deflated 39%)
2023-01-11T22:55:29.6484967Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225315.xml (deflated 39%)
2023-01-11T22:55:29.6485703Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225324.xml (deflated 38%)
2023-01-11T22:55:29.6486429Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225332.xml (deflated 39%)
2023-01-11T22:55:29.6487164Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225341.xml (deflated 39%)
2023-01-11T22:55:29.6487904Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225354.xml (deflated 39%)
2023-01-11T22:55:29.6488644Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225402.xml (deflated 39%)
2023-01-11T22:55:29.6489363Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225411.xml (deflated 39%)
2023-01-11T22:55:29.6490099Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225419.xml (deflated 39%)
2023-01-11T22:55:29.6490928Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225428.xml (deflated 39%)
2023-01-11T22:55:29.6491672Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225436.xml (deflated 38%)
2023-01-11T22:55:29.6492390Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225444.xml (deflated 39%)
2023-01-11T22:55:29.6493126Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225453.xml (deflated 39%)
2023-01-11T22:55:29.6493860Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225507.xml (deflated 39%)
2023-01-11T22:55:29.6494630Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20230111225515.xml (deflated 40%)
2023-01-11T22:55:29.6517786Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T22:55:29.6518182Z # Remove any previous test reports if they exist
2023-01-11T22:55:29.6518502Z rm -f usage-log-*.zip
2023-01-11T22:55:29.6518877Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2023-01-11T22:55:29.6519256Z # so check to see if the file exists first
2023-01-11T22:55:29.6520069Z if [ -f 'usage_log.txt' ]; then
2023-01-11T22:55:29.6520414Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2023-01-11T22:55:29.6520690Z fi
2023-01-11T22:55:29.6532857Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:55:29.6533154Z env:
2023-01-11T22:55:29.6533381Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:29.6533646Z   GPU_FLAG: --gpus all
2023-01-11T22:55:29.6534005Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:29.6534465Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501
2023-01-11T22:55:29.6534807Z ##[endgroup]
2023-01-11T22:55:29.7424444Z   adding: usage_log.txt (deflated 95%)
2023-01-11T22:55:29.7472172Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:55:29.7472463Z with:
2023-01-11T22:55:29.7472829Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T22:55:29.7473143Z   retention-days: 14
2023-01-11T22:55:29.7473408Z   if-no-files-found: warn
2023-01-11T22:55:29.7473662Z   path: test-jsons-*.zip
2023-01-11T22:55:29.7473912Z   name: artifact
2023-01-11T22:55:29.7474161Z   s3-bucket: gha-artifacts
2023-01-11T22:55:29.7474404Z   region: us-east-1
2023-01-11T22:55:29.7474631Z env:
2023-01-11T22:55:29.7474865Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:29.7475111Z   GPU_FLAG: --gpus all
2023-01-11T22:55:29.7475465Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:29.7475813Z ##[endgroup]
2023-01-11T22:55:30.1935490Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:55:30.1936386Z With the provided path, there will be 1 file uploaded
2023-01-11T22:55:30.1936759Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T22:55:30.1948240Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501.zip
2023-01-11T22:55:30.3841653Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501.zip
2023-01-11T22:55:30.4003034Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:55:30.4003326Z with:
2023-01-11T22:55:30.4003590Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T22:55:30.4003887Z   retention-days: 14
2023-01-11T22:55:30.4004154Z   if-no-files-found: error
2023-01-11T22:55:30.4004415Z   path: test-reports-*.zip
2023-01-11T22:55:30.4004666Z   name: artifact
2023-01-11T22:55:30.4004913Z   s3-bucket: gha-artifacts
2023-01-11T22:55:30.4005155Z   region: us-east-1
2023-01-11T22:55:30.4005383Z env:
2023-01-11T22:55:30.4005619Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:30.4006000Z   GPU_FLAG: --gpus all
2023-01-11T22:55:30.4006363Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:30.4006707Z ##[endgroup]
2023-01-11T22:55:30.8512162Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:55:30.8513071Z With the provided path, there will be 1 file uploaded
2023-01-11T22:55:30.8513449Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T22:55:30.8525194Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501.zip
2023-01-11T22:55:31.1074216Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501.zip
2023-01-11T22:55:31.1241796Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:55:31.1242087Z with:
2023-01-11T22:55:31.1242366Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T22:55:31.1242648Z   retention-days: 14
2023-01-11T22:55:31.1242915Z   if-no-files-found: ignore
2023-01-11T22:55:31.1243196Z   path: usage-log-*.zip
2023-01-11T22:55:31.1243425Z   name: artifact
2023-01-11T22:55:31.1243673Z   s3-bucket: gha-artifacts
2023-01-11T22:55:31.1243932Z   region: us-east-1
2023-01-11T22:55:31.1244141Z env:
2023-01-11T22:55:31.1244375Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:31.1244647Z   GPU_FLAG: --gpus all
2023-01-11T22:55:31.1244987Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:31.1245330Z ##[endgroup]
2023-01-11T22:55:31.5724809Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:55:31.5725639Z With the provided path, there will be 1 file uploaded
2023-01-11T22:55:31.5726012Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T22:55:31.5737233Z Starting upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501.zip
2023-01-11T22:55:31.7488042Z Finished upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589291501.zip
2023-01-11T22:55:31.7678833Z ##[group]Run # shellcheck disable=SC2156
2023-01-11T22:55:31.7679201Z # shellcheck disable=SC2156
2023-01-11T22:55:31.7680054Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2023-01-11T22:55:31.7693691Z shell: /usr/bin/bash -e {0}
2023-01-11T22:55:31.7693931Z env:
2023-01-11T22:55:31.7694176Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:31.7694508Z   GPU_FLAG: --gpus all
2023-01-11T22:55:31.7694856Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:31.7695204Z ##[endgroup]
2023-01-11T22:55:32.0966131Z ##[group]Run set -x
2023-01-11T22:55:32.0966428Z set -x
2023-01-11T22:55:32.0966712Z python3 -m pip install -r requirements.txt
2023-01-11T22:55:32.0967052Z python3 -m pip install boto3==1.19.12
2023-01-11T22:55:32.0967447Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T22:55:32.0980274Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:55:32.0980551Z env:
2023-01-11T22:55:32.0980795Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:55:32.0981062Z   GPU_FLAG: --gpus all
2023-01-11T22:55:32.0981403Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:55:32.0981767Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T22:55:32.0982031Z   BRANCH: pull/91627
2023-01-11T22:55:32.0982275Z   TEST_CONFIG: distributed
2023-01-11T22:55:32.0982524Z   SHARD_NUMBER: 2
2023-01-11T22:55:32.0982837Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T22:55:32.0983171Z   PR_NUMBER: 91627
2023-01-11T22:55:32.0983437Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T22:55:32.0983705Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T22:55:32.0984019Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T22:55:32.0984296Z   TAG: 
2023-01-11T22:55:32.0984508Z   WORKFLOW_ID: 3896099317
2023-01-11T22:55:32.0984936Z   GITHUB_TOKEN: ***
2023-01-11T22:55:32.0985326Z   GHA_WORKFLOW_JOB_ID: 10589291501
2023-01-11T22:55:32.0985569Z ##[endgroup]
2023-01-11T22:55:32.1014556Z + python3 -m pip install -r requirements.txt
2023-01-11T22:55:32.3906606Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:55:32.4723761Z Collecting astunparse
2023-01-11T22:55:32.4871692Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2023-01-11T22:55:32.5226071Z Collecting expecttest
2023-01-11T22:55:32.5278604Z   Downloading expecttest-0.1.4-py3-none-any.whl (6.5 kB)
2023-01-11T22:55:32.5693795Z Collecting future
2023-01-11T22:55:32.5763748Z   Downloading future-0.18.2.tar.gz (829 kB)
2023-01-11T22:55:34.5695123Z Collecting hypothesis
2023-01-11T22:55:34.5767591Z   Downloading hypothesis-6.62.0-py3-none-any.whl (399 kB)
2023-01-11T22:55:35.2792530Z Collecting numpy
2023-01-11T22:55:35.2848406Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2023-01-11T22:55:35.7144139Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2023-01-11T22:55:35.8362710Z Collecting pyyaml
2023-01-11T22:55:35.8426455Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2023-01-11T22:55:35.8651127Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2023-01-11T22:55:35.8822506Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2023-01-11T22:55:35.9448991Z Collecting six
2023-01-11T22:55:35.9489521Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2023-01-11T22:55:35.9856501Z Collecting types-dataclasses
2023-01-11T22:55:35.9913643Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2023-01-11T22:55:36.0382159Z Collecting typing_extensions
2023-01-11T22:55:36.0427899Z   Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
2023-01-11T22:55:36.0996854Z Collecting sympy
2023-01-11T22:55:36.1088720Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2023-01-11T22:55:36.3237118Z Collecting filelock
2023-01-11T22:55:36.3280041Z   Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB)
2023-01-11T22:55:36.4232616Z Collecting networkx
2023-01-11T22:55:36.4314814Z   Downloading networkx-2.6.3-py3-none-any.whl (1.9 MB)
2023-01-11T22:55:36.5588872Z Collecting jinja2
2023-01-11T22:55:36.5636861Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2023-01-11T22:55:36.6609023Z Collecting wheel<1.0,>=0.23.0
2023-01-11T22:55:36.6650957Z   Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
2023-01-11T22:55:36.7115357Z Collecting sortedcontainers<3.0.0,>=2.1.0
2023-01-11T22:55:36.7157970Z   Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
2023-01-11T22:55:36.7674174Z Collecting attrs>=19.2.0
2023-01-11T22:55:36.7715001Z   Downloading attrs-22.2.0-py3-none-any.whl (60 kB)
2023-01-11T22:55:36.8492773Z Collecting exceptiongroup>=1.0.0; python_version < "3.11"
2023-01-11T22:55:36.8533347Z   Downloading exceptiongroup-1.1.0-py3-none-any.whl (14 kB)
2023-01-11T22:55:36.8640856Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2023-01-11T22:55:36.8656425Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2023-01-11T22:55:36.8684691Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.14)
2023-01-11T22:55:36.8907103Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.12.7)
2023-01-11T22:55:36.9190491Z Collecting mpmath>=0.19
2023-01-11T22:55:36.9249813Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2023-01-11T22:55:37.0772114Z Collecting MarkupSafe>=2.0
2023-01-11T22:55:37.0814078Z   Downloading MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
2023-01-11T22:55:37.0916804Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2023-01-11T22:55:37.2811263Z Installing collected packages: six, wheel, astunparse, expecttest, future, sortedcontainers, attrs, exceptiongroup, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy, filelock, networkx, MarkupSafe, jinja2
2023-01-11T22:55:37.3246357Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2023-01-11T22:55:37.3247004Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T22:55:37.3633299Z     Running setup.py install for future: started
2023-01-11T22:55:38.0470438Z     Running setup.py install for future: finished with status 'done'
2023-01-11T22:55:38.3614304Z   WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2023-01-11T22:55:38.3614960Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T22:55:40.3707260Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2023-01-11T22:55:40.3707989Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T22:55:49.4654658Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2023-01-11T22:55:50.7465858Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T22:55:50.7467007Z Successfully installed MarkupSafe-2.1.1 astunparse-1.6.3 attrs-22.2.0 exceptiongroup-1.1.0 expecttest-0.1.4 filelock-3.9.0 future-0.18.2 hypothesis-6.62.0 jinja2-3.1.2 mpmath-1.2.1 networkx-2.6.3 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.4.0 wheel-0.38.4
2023-01-11T22:55:50.8228894Z + python3 -m pip install boto3==1.19.12
2023-01-11T22:55:51.1227686Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:55:52.1414806Z Collecting boto3==1.19.12
2023-01-11T22:55:52.1576867Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2023-01-11T22:55:53.3377754Z Collecting botocore<1.23.0,>=1.22.12
2023-01-11T22:55:53.3485956Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2023-01-11T22:55:53.6066630Z Collecting s3transfer<0.6.0,>=0.5.0
2023-01-11T22:55:53.6107876Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2023-01-11T22:55:53.6674663Z Collecting jmespath<1.0.0,>=0.7.1
2023-01-11T22:55:53.6714919Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2023-01-11T22:55:53.6854937Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.14)
2023-01-11T22:55:53.7600772Z Collecting python-dateutil<3.0.0,>=2.1
2023-01-11T22:55:53.7646565Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2023-01-11T22:55:53.7848315Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2023-01-11T22:55:54.0021810Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2023-01-11T22:55:54.9185150Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2023-01-11T22:55:54.9751478Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T22:57:24.2722934Z [scribe] Scribe access token not provided, sending report via boto3...
2023-01-11T22:57:24.2723775Z ERROR ENCOUNTERED WHEN UPLOADING TO SCRIBE: {"errorMessage":"2023-01-11T22:56:55.747Z e05ded91-b991-4cfd-b981-27347b2f3f99 Task timed out after 60.01 seconds"}
2023-01-11T22:57:24.2724719Z 
2023-01-11T22:57:24.2724961Z ----- Historic stats comparison result ------
2023-01-11T22:57:24.2725167Z 
2023-01-11T22:57:24.2728113Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2023-01-11T22:57:24.2728679Z     commit: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T22:57:24.2728918Z 
2023-01-11T22:57:24.2729128Z Commit graph (base is most recent master ancestor with at least one S3 report):
2023-01-11T22:57:24.2729376Z 
2023-01-11T22:57:24.2732842Z     : (master)
2023-01-11T22:57:24.2733112Z     |
2023-01-11T22:57:24.2733399Z     | * 8419ddda87 (HEAD)            total time  3219.40s
2023-01-11T22:57:24.2733671Z     | |
2023-01-11T22:57:24.2733883Z     | : (2 commits)
2023-01-11T22:57:24.2734111Z     |/
2023-01-11T22:57:24.2734927Z     * db2a237763 (base)  18 reports, total time  5298.55s ± 3363.99s
2023-01-11T22:57:24.2845132Z     * 2b0abd4ce3         18 reports, total time  5375.52s ± 3469.84s
2023-01-11T22:57:24.2845900Z     * f7939b21e1         48 reports, total time  4156.05s ± 3748.80s
2023-01-11T22:57:24.2846353Z     * cb3204823e         18 reports, total time  5386.78s ± 3494.12s
2023-01-11T22:57:24.2846749Z     * 6e236553f5         18 reports, total time  5359.08s ± 3471.03s
2023-01-11T22:57:24.2847150Z     * cce577b391         18 reports, total time  5383.89s ± 3498.12s
2023-01-11T22:57:24.2847570Z     * fae821c2f1         18 reports, total time  5181.34s ± 3399.30s
2023-01-11T22:57:24.2848007Z     * 0c3659586d         18 reports, total time  5181.30s ± 3409.47s
2023-01-11T22:57:24.2848427Z     * 122245985a         18 reports, total time  5175.63s ± 3372.26s
2023-01-11T22:57:24.2848821Z     * b797a24259         18 reports, total time  5151.22s ± 3376.21s
2023-01-11T22:57:24.2849097Z     |
2023-01-11T22:57:24.2849310Z     :
2023-01-11T22:57:24.2849427Z 
2023-01-11T22:57:24.2849598Z Removed  (across 1482 suites)     0 tests, totaling     0.00s
2023-01-11T22:57:24.2849964Z Modified (across    0 suites)     0 tests, totaling     0.00s
2023-01-11T22:57:24.2850313Z Added    (across   75 suites)   898 tests, totaling +4090.40s
2023-01-11T22:57:24.3373027Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2023-01-11T22:57:24.3373375Z with:
2023-01-11T22:57:24.3373591Z env:
2023-01-11T22:57:24.3373830Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:57:24.3374080Z   GPU_FLAG: --gpus all
2023-01-11T22:57:24.3374441Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:57:24.3374788Z ##[endgroup]
2023-01-11T22:57:24.3396682Z ##[group]Run set -eou pipefail
2023-01-11T22:57:24.3396990Z set -eou pipefail
2023-01-11T22:57:24.3397241Z 
2023-01-11T22:57:24.3397560Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2023-01-11T22:57:24.3397883Z for _ in $(seq 1440); do
2023-01-11T22:57:24.3398182Z     # Break if no ssh session exists anymore
2023-01-11T22:57:24.3398492Z     if [ "$(who)" = "" ]; then
2023-01-11T22:57:24.3398727Z       break
2023-01-11T22:57:24.3398988Z     fi
2023-01-11T22:57:24.3399228Z     echo "."
2023-01-11T22:57:24.3399885Z     sleep 5
2023-01-11T22:57:24.3400268Z done
2023-01-11T22:57:24.3413815Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:57:24.3414108Z env:
2023-01-11T22:57:24.3414331Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:57:24.3414598Z   GPU_FLAG: --gpus all
2023-01-11T22:57:24.3414956Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:57:24.3415284Z ##[endgroup]
2023-01-11T22:57:24.3447046Z Holding runner for 2 hours until all ssh sessions have logged out
2023-01-11T22:57:24.3531538Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T22:57:24.3531967Z # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T22:57:24.3532443Z # shellcheck disable=SC2046
2023-01-11T22:57:24.3532761Z docker stop $(docker ps -q) || true
2023-01-11T22:57:24.3533065Z # Prune all of the docker images
2023-01-11T22:57:24.3533367Z docker system prune -af
2023-01-11T22:57:24.3545687Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:57:24.3545969Z env:
2023-01-11T22:57:24.3546216Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:57:24.3546486Z   GPU_FLAG: --gpus all
2023-01-11T22:57:24.3546830Z   DOCKER_CONTAINER_ID: f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:57:24.3547179Z ##[endgroup]
2023-01-11T22:57:24.8677708Z f50ab5966376
2023-01-11T22:57:25.9574724Z Deleted Containers:
2023-01-11T22:57:25.9575112Z f50ab59663760b3879a1b4b07fdb94fc7a430ae47bba938264724dcd9a8571bc
2023-01-11T22:57:25.9575339Z 
2023-01-11T22:57:30.8609271Z Deleted Images:
2023-01-11T22:57:30.8610762Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T22:57:30.8612545Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:866df6c1171dbe014496717cf2080d6cc72ca611a4e8146525c9ef09640c8ba4
2023-01-11T22:57:30.8613156Z deleted: sha256:09e297797cd8c095524ba49e041c45e57bf05ba16719f01e7240e8549da5beba
2023-01-11T22:57:30.8613546Z deleted: sha256:6d5f0082fbf8c3b01e49961283f44105a5bb12616f0073762021db97f20a16a5
2023-01-11T22:57:30.8614040Z deleted: sha256:19c574c96e47e3d16cc51cc088fed6cef16eaa2174e667ba0b395ac0e3b989bb
2023-01-11T22:57:30.8614930Z deleted: sha256:4fa7af758c23581dbcc2bd92defdc5fea97c8671fa67bbc30888ffbbf96c49a6
2023-01-11T22:57:30.8615496Z deleted: sha256:e68331e1b0f863bbdbd445ed8475d59d2234c9659264a4e49b7b096311445aee
2023-01-11T22:57:30.8615897Z deleted: sha256:69d886418998cd8758c555ed219fca3a457539e9d6f62c41c9664a80c82c4036
2023-01-11T22:57:30.8616316Z deleted: sha256:2368b1fdf0235d155eaa47e75ff379cff06cc82d63e03886a57363b8092d3c83
2023-01-11T22:57:30.8616772Z deleted: sha256:651c1e1b625aeaf8fa65e96ef11487e650a7a6d00ba8ec9fdfe7e89e762dc5c4
2023-01-11T22:57:30.8617407Z deleted: sha256:7b73f298df08c4b3aa849b55bb5e73ded619cb5b786fddcc21ece3c0b3887038
2023-01-11T22:57:30.8617869Z deleted: sha256:716f28b4b433958e5d1b9839a20dd22f9986f9c1b42fb95552f13d6bfd291efe
2023-01-11T22:57:30.8618270Z deleted: sha256:e4495993276176c504228e66b2fd6f348523c5af66f9292b2f7ea12acefcd606
2023-01-11T22:57:30.8618684Z deleted: sha256:d6f5fbd8782783697c73f4bcbce91a05e126575f72f0f58e1e9465aa57640a92
2023-01-11T22:57:30.8619091Z deleted: sha256:0b7725c897ee2681e3b1ff00aea6c14805c8050758cfa1010f561c9713934014
2023-01-11T22:57:30.8619526Z deleted: sha256:fdd864c6750bdd24f8cfb131673c6f04087e3cbecac2c1a9b3c30fefbd6d3070
2023-01-11T22:57:30.8619993Z deleted: sha256:bfaeaf77f180f62f3994ccdc2be80dd2ef7f4d25ffc8c9497dc51e6cff69711e
2023-01-11T22:57:30.8620437Z deleted: sha256:02f5a9d8be5a1bdd5a350d4c47147fd3dd46bfcedc7637f53a8a692720381fc2
2023-01-11T22:57:30.8620902Z deleted: sha256:e22cc66e4fd2e491fb4ae8194c35d6b1789f9f5d01e1dfbedf1c266c3a1537de
2023-01-11T22:57:30.8621327Z deleted: sha256:1536d02ae84ab410916541408cf2935f122735cc5d128324f6f82fbbeb913e80
2023-01-11T22:57:30.8621746Z deleted: sha256:a8374bc83a4bf3a838aaf8ee71b4a8281ac4eef801473b007f88b2f0efadc6c6
2023-01-11T22:57:30.8622153Z deleted: sha256:9381921b1e612b2d23517d24662a39f00be43efe2412e440ef41b487a48cb389
2023-01-11T22:57:30.8622575Z deleted: sha256:ae063d6cbeae3688a5a7bb8694d431dbb9792bfb7ff2908d3b25842f9586fc8e
2023-01-11T22:57:30.8623017Z deleted: sha256:595397ba048a351c7b09c25b5eba4cbb916c2db40dd80bde4d95c2b51b766045
2023-01-11T22:57:30.8623452Z deleted: sha256:7a2da4ffb8ea2b858fcdfc92f6e640dbb6a083b39dfa5e7aa87cd1296f9314e1
2023-01-11T22:57:30.8623919Z deleted: sha256:eddd66bfdbb5913f133ffda8d967ee0235f9f121434112ea8da4cfdf5f9ebbf4
2023-01-11T22:57:30.8624359Z deleted: sha256:c9392d92ee837d52b35b41e4de67d213886d844cecd9e769d9284dc21070aee8
2023-01-11T22:57:30.8624935Z deleted: sha256:fba16b8beafe9efa854d93f0e92718750ec97ced755ace0f6f51bbd5d1964f91
2023-01-11T22:57:30.8625348Z deleted: sha256:a09592e27d2e6896b9029f31e269787c92761fd19a48528851c48d85221cd4bc
2023-01-11T22:57:30.8625752Z deleted: sha256:8b2e3a8416af60625ccb9a8562891c1b9e85c2ee05f103b190ad5040313bf1f1
2023-01-11T22:57:30.8626185Z deleted: sha256:135caddb443044601f433d421e5c0f5d8ab02dff69cf2df9024a8ecb97c8948c
2023-01-11T22:57:30.8626574Z deleted: sha256:112b60db47585e101175390e102a66908f0f1175510c5e5d5f10b7e4e0c9769b
2023-01-11T22:57:30.8626986Z deleted: sha256:d9e5dd4e760b68190c010a9042c842afb0bcf3d4477a334d0e9c2d9c302ecb3c
2023-01-11T22:57:30.8627420Z deleted: sha256:7d37673dfb91518db6e68e9637f3db142a4eaccb9f548ab99d14ac52b3672325
2023-01-11T22:57:30.8627836Z deleted: sha256:12784d32c23b63941ebc8adf713d7167158c38c771d3b5f94506e036b6273dbd
2023-01-11T22:57:30.8628237Z deleted: sha256:395db4c8bde7e390accfa81ff84376004b78566423bd0a3bd7559e12e66759bc
2023-01-11T22:57:30.8628666Z deleted: sha256:a7aa04ee64333d427277f47b8e07dee6cc566b132a18ee166c305c890d4fd3bb
2023-01-11T22:57:30.8629090Z deleted: sha256:10a0e8d138614f29a13aecd15175039c7f86ca04fc588b439f56284b3c0e292a
2023-01-11T22:57:30.8629503Z deleted: sha256:68d97ecc8d2ec5f756b4a1b7e7451a413ed19f3d7c3ce52c81b30dc00fd96185
2023-01-11T22:57:30.8629929Z deleted: sha256:e833f7b95e0efaaf775293760365243991395e796d587deffe588e50fe7a9f1e
2023-01-11T22:57:30.8630341Z deleted: sha256:cbcd1e7502e949614d05d5d0a34316fd62665c3e030b3656d9c488cfee1eca34
2023-01-11T22:57:30.8630786Z deleted: sha256:a90ce4a75d9d408df6349607788566125c2662c603dc8f84c767b2256273ff12
2023-01-11T22:57:30.8631196Z deleted: sha256:bdee06da46fd67ff14bbc7286b40ef15174551b6452ca633be8576376a3dbddb
2023-01-11T22:57:30.8631639Z deleted: sha256:2cafe83ace87d1a83f30f8d458001f3a315a606d5251c20543cbd6604499aa73
2023-01-11T22:57:30.8632081Z deleted: sha256:cc1f7fb1208e7b05b48d3b3b2648946adaa723b10df2574c7db87ed9112b1510
2023-01-11T22:57:30.8632519Z deleted: sha256:9027ec66ecfd6490ecf157cdf665428dad438f54565d9d85263211da30e6684d
2023-01-11T22:57:30.8632944Z deleted: sha256:a27541cda46a9ede5932b1a1807360e2f6ada5bb0c30bfa3aa953d59dcc1bc5c
2023-01-11T22:57:30.8633469Z deleted: sha256:7b927de6f9fdeb74acee54e7654d04e2614a112cafb477b2433db34dcc7ebe28
2023-01-11T22:57:30.8633904Z deleted: sha256:8298c8753925ad5124b0611c4101e92ec1f877252f8320f5503ebf3e4e7e1314
2023-01-11T22:57:30.8634293Z deleted: sha256:0b741a2d83533cb3b47b912506d486fd477d8c8a1c520a0f9d7d62edfc55487d
2023-01-11T22:57:30.8634719Z deleted: sha256:9412a0a6fd6057a9939fed5beadf64568de1d230c0325a628e396f5b76444bbb
2023-01-11T22:57:30.8635177Z deleted: sha256:0a294dad6a5df3fab407cd28b31eadaf9f65a5bdbf8c7a0db349d3401b537cef
2023-01-11T22:57:30.8635629Z deleted: sha256:b42ee7d4f5715886c70d5cfef4724e889f773a20741ebc4ed9c1771eb09ad634
2023-01-11T22:57:30.8636036Z deleted: sha256:a68106c7b0e03f1e0fa9ad405ce403e2931174825bdcd8e259522ef40ec8a617
2023-01-11T22:57:30.8636460Z deleted: sha256:ad403ee05a64f55c765e665bd3f25a71650857123282dc1bad2af81f5665cda5
2023-01-11T22:57:30.8636886Z deleted: sha256:eb731dc1382c6cc80193d0454f740fc55f441c32f08cd0ce1b8784e7840e53df
2023-01-11T22:57:30.8637283Z deleted: sha256:45bbe3d22998589317c7f6c4dd591475423bb37ca9b922529c5878653483b18d
2023-01-11T22:57:30.8637514Z 
2023-01-11T22:57:30.8750152Z Total reclaimed space: 21.38GB
2023-01-11T22:57:30.8811533Z Post job cleanup.
2023-01-11T22:57:30.8854436Z Post job cleanup.
2023-01-11T22:57:31.0257235Z [command]/usr/bin/git version
2023-01-11T22:57:31.0316069Z git version 2.38.1
2023-01-11T22:57:31.0383875Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/fbcad613-b07e-47bc-a6e0-0fcc605db092' before making global git config changes
2023-01-11T22:57:31.0385757Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T22:57:31.0391076Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T22:57:31.0433601Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T22:57:31.0472590Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T22:57:31.0818891Z Entering 'android/libs/fbjni'
2023-01-11T22:57:31.0864773Z Entering 'third_party/FP16'
2023-01-11T22:57:31.0910939Z Entering 'third_party/FXdiv'
2023-01-11T22:57:31.0956340Z Entering 'third_party/NNPACK'
2023-01-11T22:57:31.1002691Z Entering 'third_party/QNNPACK'
2023-01-11T22:57:31.1048229Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T22:57:31.1093570Z Entering 'third_party/XNNPACK'
2023-01-11T22:57:31.1149209Z Entering 'third_party/benchmark'
2023-01-11T22:57:31.1194017Z Entering 'third_party/cpuinfo'
2023-01-11T22:57:31.1241599Z Entering 'third_party/cub'
2023-01-11T22:57:31.1287037Z Entering 'third_party/cudnn_frontend'
2023-01-11T22:57:31.1337244Z Entering 'third_party/cutlass'
2023-01-11T22:57:31.1388102Z Entering 'third_party/eigen'
2023-01-11T22:57:31.1434065Z Entering 'third_party/fbgemm'
2023-01-11T22:57:31.1477324Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T22:57:31.1519973Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T22:57:31.1563133Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T22:57:31.1607446Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T22:57:31.1651307Z Entering 'third_party/flatbuffers'
2023-01-11T22:57:31.1699533Z Entering 'third_party/fmt'
2023-01-11T22:57:31.1743133Z Entering 'third_party/foxi'
2023-01-11T22:57:31.1786024Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T22:57:31.1830633Z Entering 'third_party/gloo'
2023-01-11T22:57:31.1875972Z Entering 'third_party/googletest'
2023-01-11T22:57:31.1919104Z Entering 'third_party/ideep'
2023-01-11T22:57:31.1961536Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T22:57:31.2007505Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T22:57:31.2057901Z Entering 'third_party/ios-cmake'
2023-01-11T22:57:31.2101195Z Entering 'third_party/ittapi'
2023-01-11T22:57:31.2144253Z Entering 'third_party/kineto'
2023-01-11T22:57:31.2188400Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T22:57:31.2232317Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T22:57:31.2277854Z Entering 'third_party/nccl/nccl'
2023-01-11T22:57:31.2324962Z Entering 'third_party/neon2sse'
2023-01-11T22:57:31.2370363Z Entering 'third_party/nlohmann'
2023-01-11T22:57:31.2414684Z Entering 'third_party/onnx'
2023-01-11T22:57:31.2472974Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T22:57:31.2517693Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T22:57:31.2564471Z Entering 'third_party/onnx-tensorrt'
2023-01-11T22:57:31.2610134Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T22:57:31.2659470Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T22:57:31.2706326Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T22:57:31.2750311Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T22:57:31.2801314Z Entering 'third_party/pocketfft'
2023-01-11T22:57:31.2847852Z Entering 'third_party/protobuf'
2023-01-11T22:57:31.2896398Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T22:57:31.2939820Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T22:57:31.2986821Z Entering 'third_party/psimd'
2023-01-11T22:57:31.3030673Z Entering 'third_party/pthreadpool'
2023-01-11T22:57:31.3075222Z Entering 'third_party/pybind11'
2023-01-11T22:57:31.3118847Z Entering 'third_party/python-enum'
2023-01-11T22:57:31.3162633Z Entering 'third_party/python-peachpy'
2023-01-11T22:57:31.3207322Z Entering 'third_party/python-six'
2023-01-11T22:57:31.3250529Z Entering 'third_party/sleef'
2023-01-11T22:57:31.3293803Z Entering 'third_party/tbb'
2023-01-11T22:57:31.3340130Z Entering 'third_party/tensorpipe'
2023-01-11T22:57:31.3384324Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T22:57:31.3427240Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T22:57:31.3470542Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T22:57:31.3514012Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T22:57:31.3555830Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T22:57:31.3602158Z Entering 'third_party/zstd'
2023-01-11T22:57:31.3662542Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T22:57:31.3695181Z http.https://github.com/.extraheader
2023-01-11T22:57:31.3705133Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-01-11T22:57:31.3745419Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T22:57:31.4070336Z Entering 'android/libs/fbjni'
2023-01-11T22:57:31.4096122Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4129973Z Entering 'third_party/FP16'
2023-01-11T22:57:31.4155008Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4189315Z Entering 'third_party/FXdiv'
2023-01-11T22:57:31.4215344Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4248552Z Entering 'third_party/NNPACK'
2023-01-11T22:57:31.4273193Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4306966Z Entering 'third_party/QNNPACK'
2023-01-11T22:57:31.4333332Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4367546Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T22:57:31.4392469Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4427492Z Entering 'third_party/XNNPACK'
2023-01-11T22:57:31.4453607Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4499766Z Entering 'third_party/benchmark'
2023-01-11T22:57:31.4525839Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4559246Z Entering 'third_party/cpuinfo'
2023-01-11T22:57:31.4585457Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4620544Z Entering 'third_party/cub'
2023-01-11T22:57:31.4647332Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4681789Z Entering 'third_party/cudnn_frontend'
2023-01-11T22:57:31.4706930Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4747122Z Entering 'third_party/cutlass'
2023-01-11T22:57:31.4773206Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4814271Z Entering 'third_party/eigen'
2023-01-11T22:57:31.4839258Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4875552Z Entering 'third_party/fbgemm'
2023-01-11T22:57:31.4901708Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4940413Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T22:57:31.4966112Z http.https://github.com/.extraheader
2023-01-11T22:57:31.4998466Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T22:57:31.5024008Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5057889Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T22:57:31.5088663Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5121624Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T22:57:31.5145488Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5180314Z Entering 'third_party/flatbuffers'
2023-01-11T22:57:31.5205926Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5243626Z Entering 'third_party/fmt'
2023-01-11T22:57:31.5268923Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5303443Z Entering 'third_party/foxi'
2023-01-11T22:57:31.5329094Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5363293Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T22:57:31.5388945Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5424526Z Entering 'third_party/gloo'
2023-01-11T22:57:31.5451579Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5485478Z Entering 'third_party/googletest'
2023-01-11T22:57:31.5510495Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5544092Z Entering 'third_party/ideep'
2023-01-11T22:57:31.5569754Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5602805Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T22:57:31.5627271Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5662825Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T22:57:31.5687616Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5728812Z Entering 'third_party/ios-cmake'
2023-01-11T22:57:31.5754439Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5788915Z Entering 'third_party/ittapi'
2023-01-11T22:57:31.5814976Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5849144Z Entering 'third_party/kineto'
2023-01-11T22:57:31.5874639Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5908748Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T22:57:31.5933621Z http.https://github.com/.extraheader
2023-01-11T22:57:31.5966490Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T22:57:31.5990750Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6027215Z Entering 'third_party/nccl/nccl'
2023-01-11T22:57:31.6052575Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6087581Z Entering 'third_party/neon2sse'
2023-01-11T22:57:31.6112730Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6150282Z Entering 'third_party/nlohmann'
2023-01-11T22:57:31.6176144Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6212252Z Entering 'third_party/onnx'
2023-01-11T22:57:31.6237423Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6284017Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T22:57:31.6308517Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6342278Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T22:57:31.6368174Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6403451Z Entering 'third_party/onnx-tensorrt'
2023-01-11T22:57:31.6428318Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6461304Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T22:57:31.6485745Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6523454Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T22:57:31.6547739Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6582628Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T22:57:31.6607911Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6640777Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T22:57:31.6665401Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6704478Z Entering 'third_party/pocketfft'
2023-01-11T22:57:31.6730862Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6764809Z Entering 'third_party/protobuf'
2023-01-11T22:57:31.6789872Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6828013Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T22:57:31.6852601Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6885269Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T22:57:31.6909548Z http.https://github.com/.extraheader
2023-01-11T22:57:31.6945844Z Entering 'third_party/psimd'
2023-01-11T22:57:31.6971737Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7006373Z Entering 'third_party/pthreadpool'
2023-01-11T22:57:31.7031590Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7064596Z Entering 'third_party/pybind11'
2023-01-11T22:57:31.7090593Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7125881Z Entering 'third_party/python-enum'
2023-01-11T22:57:31.7151161Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7186760Z Entering 'third_party/python-peachpy'
2023-01-11T22:57:31.7212668Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7247054Z Entering 'third_party/python-six'
2023-01-11T22:57:31.7271682Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7306544Z Entering 'third_party/sleef'
2023-01-11T22:57:31.7332343Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7367839Z Entering 'third_party/tbb'
2023-01-11T22:57:31.7393488Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7430234Z Entering 'third_party/tensorpipe'
2023-01-11T22:57:31.7457028Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7491323Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T22:57:31.7516371Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7551126Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T22:57:31.7576792Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7609690Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T22:57:31.7634034Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7667340Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T22:57:31.7691830Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7724146Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T22:57:31.7749759Z http.https://github.com/.extraheader
2023-01-11T22:57:31.7788748Z Entering 'third_party/zstd'
2023-01-11T22:57:31.7814294Z http.https://github.com/.extraheader
2023-01-11T22:57:31.8117113Z Cleaning up orphan processes